Unlocking the Power of Data: Streamlined ETL Solutions for the Digital Anthroposphere
Ali Asghari1, Josh Clough1, Annie Wu1, Tim Leonard1, Jason Kreitner1, Ivan Widjaja1 1Australian Urban Research Infrastructure Network (AURIN), Melbourne, Victoria, Australia
Abstract
The National Collaborative Research Infrastructure Strategy (NCRIS) funds the Australian Urban Research Infrastructure Network (AURIN) to provide national research infrastructure which plays a key role in supporting Australia’s response to the rising challenges of climate change, energy transition, and demographic change. AURIN facilitates this through the Urban Data as a Service (UDaaS) platform which aims to seamlessly empower researchers and Urban Digital Twin (UDT) efforts across Australia with high-value data to inform planning, policy and decision making.
The UDaaS platform provides users with data that is curated and standardised to reduce the costs of transactions for the development and management of the UDTs. AURIN utilises and continues to develop an Extract, Transform and Load (ETL) data pipeline focusing on enabling the creation of data products in a reproducible, atomic, automated, and flexible manner. Distributed infrastructure acts as the basis to allow for the flexibility and scalability of the data pipelines; and Directed Acyclic Graphs (DAGs) provide control for actions made to the data. The AURIN ETL pipeline allows ingestion, curation, and delivery of data from any data provider to any data consumer in a robust and streamlined way. It is designed flexible enough to seamlessly incorporate new and emerging data formats, empowering users to harness the full potential of data from various sources in their analyses and applications.
AURIN’s commitment to innovative technologies and streamlined processes makes it a crucial resource for urban research and evidence-based decision-making to tackle critical urban challenges.
Biography
Dr Ali Asghari is a Data Engineer at AURIN, contributing to the development of robust data pipelines and scalable data architectures. Ali holds a PhD in Geomatics Engineering from the University of Melbourne. His research focused on leveraging 3D data for urban land administration, developing novel geometrical methods to validate complex ownership settings in multi-story buildings. He also has extensive expertise in university teaching and has published several papers in the areas of 3D cadastre, 3D spatial data validation, and land-use change modeling.
ORCID: https://orcid.org/0000-0001-8762-2803