Koh P, Newman P
1Atlas of Living Australia, CSIRO, Canberra, Australia, 2Atlas of Living Australia, CSIRO, Melbourne, Australia
Biography:
Patricia Koh has worked at the Atlas of Living Australia (ALA) since 2017 and transitioned to the Data Team in late 2020 as a Data Engineer/Analyst. She is responsible for developing and automating large-scale data ingestion pipelines and workflow orchestration systems. Patricia became an AWS Certified Solutions Architect in 2024 and is passionate about building scalable and resilient data platforms.
Abstract:
The Atlas of Living Australia (ALA) manages more than 180 million of Australian biodiversity occurrence records contributed by museums, herbaria, government agencies, citizen science projects, and research. As data volumes continue to grow, processing these records efficiently and reliably becomes increasingly challenging.
This lightning talk shares how ALA's ingestion platform addresses these challenges through a cloud-native, distributed architecture via elastic cloud services. The system processes large datasets at scale, running ingestion, validation, transformation, and indexing workflows in parallel.
The presentation will provide an overview of the platform's technical architecture, the design decisions that support scalability and resilience, and lessons learned from operating a national biodiversity data infrastructure that continues to grow in both size and complexity.
