Lahur P1
1Csiro, Clayton, Australia
Biography:
The author has been working in CSIRO for more than ten years as Research Software Engineer. He has been developing and deploying scientific software in various research projects in CSIRO.
Abstract:
The Australian Square Kilometre Array Pathfinder (ASKAP) is a radio telescope array located at Murchison Radio-astronomy Observatory (MRO) in Western Australia. It’s part of Australia Telescope National Facility (ATNF), which is operated and managed by the Space and Astronomy research unit of CSIRO. Data from the antenna array is ingested and processed using ASKAPsoft, which is a suite of software developed in-house. It’s designed to run as parallel processes on high-performance computers (HPC) and optimised for computing facilities in Pawsey.
ASKAPsoft, like other scientific software in this facility, relies on container technology for its deployment. This allows developers to work on their local machine using Docker container, then pull it to HPC as Singularity container for the actual runs. For end-users, the process of acquiring and using ASKAPsoft becomes very simple. For developers, this offers an effective means of dealing with complex dependencies. Additionally, we also use other tools to combat dependency issues that cannot be covered by container alone. One of such tools is Spack, which is a package manager for HPC. However, the need to deal with multiple tools, each with their own unique requirements and issues, has introduced yet another complication to our development and deployment efforts.
In this presentation, we would like to share a summary of our deployment problems, strategy and techniques to deal with them, as well as the lessons learned along the way.
