Mr Tommy Gatti1, Mr Harshula Jayasuriya1, Dr Aidan Heerdegen1
1ACCESS-NRI, Australia
Biography:
Tommy is a Senior Research Software Engineer for the Model Release Team at ACCESS-NRI. He's a multidisciplinary programmer who enjoys developing CI/CD infrastructure and technical pathfinding in different domains. In previous industry roles he has been a driver developer and member of a third level support team.
Abstract:
Developers are human, and to err is human – and this erring is a barrier to providing reproducible builds and clear provenance chains for climate models and software builds. Often, questions are raised regarding the links between how binaries and components are changed, and where those binaries are located. These essential questions are ripe for addressing not by a human, but by a machine – through a continuous integration and deployment process using a High-Performance Computing (HPC) package manager with provenance in mind.
Using GitHub Actions and the build-from-source package manager Spack, we developed generic, automated workflows that validate standardised manifests and creates clear links between model changes (for example, source code, dependencies or build options), and the deployment of models onto HPCs. This workflow enables developers proposing changes via Pull Requests (PRs) to use a standardised manifest to specify versions, configuration and variation on dependencies and libraries of the final binaries. This manifest is used by an instance of Spack on a HPC to cache and deploy software. Post-install, the PR reflects the status of the deployment, information on how to access the binaries, and provides artifacts that enable future installations to be reproducible, replicable and reliable – a full chain of provenance.
At ACCESS-NRI, these automated workflows provide a consistent provenance chain for complex software, quicker PR feedback cycles, and leaves our developers to do what they do best. They were developed to support climate models – but the methodology is applicable to a wide range of software.