Transport, movement, processing and management of big data in microscopy: the Australian Characterisation Commons at Scale project

Dr David Poger¹, Mr Chris Myers², Dr Hoang Nguyen³, Mr Jay van Schyndel⁴, Mr Joshua Silver⁵

¹Microscopy Australia, Sydney, Australia, ²AARNet, Melbourne, Australia, ³The University of Queensland, St Lucia, Australia, ⁴Monash University, Clayton, Australia, ⁵University of Wollongong, Keiraville, Australia

Thanks to innovations and breakthroughs in instrumentation in electron and correlative microscopy, a single experiment can now generate data at unprecedented resolution, scale and speed that are routinely from tens of gigabytes to a few terabytes in size. The ever-increasing amounts of data produced have posed substantial challenges as to how workflows can be organised and managed from the point of data capture through to storage and archival. The Australian Characterisation Commons at Scale (ACCS) project aims to develop and deploy tools and services as well as develop and implement procedures and guidelines to address those challenges. In this presentation, I will outline achievements in five key areas: 1. Data transport with the deployment of Globus at several universities in partnership with AARNet for fast, secure and reliable transfer of large data volumes; 2. Data movement with the optimisation and automation of workflows from the point of data capture to storage; 3. Data ingestion and management with the development of the Clowder-based data management system and repository Pitschi; 4. Data processing with the development of the Electron Microscopy Data Processing Portal for the national microscopy community; and 5. Data description, retention and disposal with the development of guidelines and the identification of institutional barriers in relation to large-scale, automated collection of metadata at microscopy facilities and the short- and long-term storage of big data. Overall, ACCS will support and uplift microscopy facilities, promote sustainable and good practices compatible with the FAIR data principles, and deliver value to researchers.

Biography:

Dr David Poger (https://orcid.org/0000-0001-8794-5688) was awarded his PhD from Joseph Fourier University, Grenoble (France) in 2005. He then moved to Australia where he worked as a Research Fellow at The University of Queensland. His research studied the structure and dynamics of cytokine receptors, biological membranes and antimicrobials using computer simulation. In 2020, David joined Microscopy Australia as Research Data Manager. He works with microscopy facilities across the country to assist them in data management and develop good practices. He is the national lead for the work package on big-data microscopy in the Australian Characterisation Commons at Scale (ACCS) project.