GBR modelling in the cloud: the architecture of eReefs RECOM2
Sharon Tickell1, Farhan Rizwi2, Jack Beardsley2, Tim Heap2, Matthew Wildie1, Daniel Wild2 1CSIRO Environment, Brisbane, QLD, Australia2CSIRO Environment, Hobart, TAS, Australia
Abstract
The eReefs RElocatable COastal Modelling service (RECOM) is a web-based application that allows non-specialist researchers and managers to configure and run high-resolution hydrodynamic and biogeochemical models for custom coastal locations or individual reefs within the Great Barrier Reef (GBR).
eReefs RECOM2 was released in May 2023 as a ground-up rewrite of the original eReefs RECOM1 platform. RECOM2 is API-driven, containerised, cloud-agnostic, hosted with a public cloud provider, and can be easily scaled-up in response to demand.
The public cloud deployment posed some interesting architectural challenges related to ensuring the models could access and subset the large (tens of TB) forcing datasets which are needed to allow RECOM models to be created for anywhere on the GBR when the model software was written assuming that forcing data would always be on a local posix filesystem.
Our solution involves object-storage systems, a Lustre filesystem cache, Python intake catalogs and Snakemake workflow templates. It not only makes the at-scale, on-demand operation of eReefs RECOM2 possible, but also had the happy side-effect that a RECOM model configured via the web user interface can be transferred to a HPC environment and run offline by power users that want to extend our default model configuration.
This poster shows our technology stack and how it works in practice.
Biography
Sharon is a software engineer who has been involved with the design, deployment, and maintenance of the CSIRO eReefs data and visualisation components since 2013, and with eReefs RECOM since 2021. She currently leads the Coastal Informatics team in the CSIRO Environment business unit and is involved with the ongoing operation of multiple eResearch platforms including eReefs, OzCoasts, and the RRAP Modelling and Decision Support information System