Dr Peter Blain1, Philip Bohm1, Leigh Gordon1, Craig Jones1, Alex McKeown1, Cameron Moloney1, George Sattler1, Angus Scheibner1
IMOS-AODN, Hobart, Australia, peter.blain@utas.edu.au, bohm@utas.edu.au leigh.gordon@utas.edu.au, craig.jones@utas.edu.au, alex.mckeown@utas.edu.au cameron.moloney@utas.edu.au, george.sattler@utas.edu.au, angus.scheibner@utas.edu.au
BACKGROUND
The Integrated Marine Observing System (IMOS) is an Australian Government infrastructure project. IMOS is an integrated national array of observing equipment that monitors the open oceans and coastal marine environment around Australia, covering physical, chemical and biological variables. All IMOS data is freely and openly available for the benefit of Australian marine and climate science. IMOS observations are guided by societal needs for improved ocean information, and focused through science planning undertaken collaboratively across the Australian marine and climate science community. The IMOS information infrastructure is built on open source applications such as GeoNetwork and GeoServer, extended by open source projects governed by IMOS.
The IMOS information infrastructure has been leveraged to build the Australian Ocean Data Network (AODN), which is an interoperable online network of marine and coastal data resources, including the Integrated Marine Observing System (IMOS), data from the six (6) Commonwealth agencies – Commonwealth Science and Industrial Research Organisation (CSIRO), Bureau of Meteorology (BOM), Royal Australian Navy (RAN), Australian Institute of Marine Science (AIMS), Geoscience Australia (GA) and the Australian Antarctic Division (AAD) – and additional data from universities, state based agencies and the private sector. The AODN provides discovery and access services, primarily through the AODN Portal. The marine data collections published in the AODN Portal are wide ranging, and all data collections are available to the public. Data covers approximately two hundred parameters in different ocean environments collected from ocean-going ships, robots, moorings and other platforms. Contributing organisations publish data in the AODN by providing ISO 19115 compliant metadata marked up with OGC web compliant services. The metadata is harvested from contributors and stored in a GeoNetwork instance that backs the AODN portal. The AODN portal is a consumer of web services including OGC CSW which is used to query the catalogue. The AODN portal also consumes web services defined in the metadata and it uses these to provide visualisation, subsetting, and data download to end users.
CLOUDIFICATION
IMOS began running its information infrastructure in the cloud at the inception of NeCTAR in 2011. In 2015 IMOS extended its move to the cloud by hosting core production systems at AWS, while continuing to use NeCTAR for research and development. The initial migration to the cloud was a classic “lift and shift” operation. This task was made relatively easy thanks to configuration management tools such as Chef. The result of this migration did not, however, yield a truly cloud-optimised architecture. The architecture looked more like a traditional physical server based design that happened to be running in the cloud. Over the last two years IMOS has transformed its infrastructure to leverage the advantages of cloud computing. This presentation discusses the architectural design patterns and cloud computing services that IMOS has put into effect. These include serverless architectures for big data aggregations, and infrastructure-as-code approaches for spinning up fully configured test and dev replicas of complex production environments.
SERVERLESS ARCHITECTURE
Serverless computing is beneficial in several situations, including when compute utilisation is lumpy. IMOS has such a use case related to the subsetting and aggregation of large gridded geospatial data collections. A single processing job may take an hour or more to complete, followed by several hours of no jobs at all. The serverless approach chosen by IMOS can run aggregation workloads at any scale. It automatically provisions compute resources without the need to install or manage the host framework, which has led to a faster and more reliable service for end users.
INFRASTRUCTURE-as-code
The main IMOS software stack comprises open source applications such as GeoServer, GeoNetwork, THREDDS, the AODN Geospatial Portal, Talend as well as object storage and a relational database. IMOS requires multiple instances of this stack for use cases such as production, testing, development, and for demonstration. The stacks must be quick to run up and tear down, and each component must scale automatically without limit. This presentation includes a discussion of how this was achieved at IMOS.
Biography:
Alex McKeown is a Software Engineer at the Australian Ocean Data Network (AODN) a facility of the Integrated Marine Observing System (IMOS). He holds a Bachelor of Computing (Honours) from the University of Tasmania and has a background of working with Sensor Web technologies.