Transport and management of data generated by big-data electron microscopy instruments: the Australian Characterisation Commons at Scale project
Dr David Poger1, Dr Hoang Nguyen2, Jay van Schyndel3, Joshua Silver4, Chris Myers5, Prof Wojtek Goscinski6
1Microscopy Australia, Sydney, Australia
2The University of Queensland, Brisbane, Australia
3Monash University, Melbourne, Australia
4University of Wollongong, Wollongong, Australia
5AARNet, Melbourne, Australia
6National Imaging Facility, Melbourne, Australia
Over the past years, Australian microscopy facilities have invested in new-generation electron and correlative microscopes. Although the considerable technological progress in instrumentation has paved the way for unprecedented scientific advances, the ever-increasing amounts of data produced have posed substantial challenges as to how workflows can be organised and managed from the point of data capture through to storage. The Australian Characterisation Commons at Scale (ACCS) project aims to develop and deploy tools and services as well as develop and implement procedures and guidelines to address those challenges. In this presentation, I will show recent outcomes of the ACCS project in data transport and data management. Specifically, after a testing phase, the data-transport service Globus has been deployed at several universities in partnership with AARNet. Globus enables fast, secure and reliable transfer of large data volumes between distant sites. A research data management system called Pitschi and based on Clowder, an open-source customisable data management framework, has been developed at The University of Queensland to ingest and manage data from microscopy instruments. Finally, I will describe a first-of-its-kind report on the informatics landscape and its challenges at national and international microscopy facilities. General trends, tools, procedures, gaps and challenges across all or most of the facilities were identified in four key areas: data movement, processing, management and orchestration. The report provides a roadmap for future software and hardware developments and outlines good practices. Overall, the ACCS project will support and uplift microscopy facilities nationally, promote collaboration and deliver value to researchers.
Dr David Poger (https://orcid.org/0000-0001-8794-5688) was awarded his PhD from Joseph Fourier University, Grenoble (France) in 2005. He then moved to Australia where he worked as a Research Fellow at The University of Queensland. His research studied the structure and dynamics of cytokine receptors, biological membranes and antimicrobials using computer simulation. In 2020, David joined Microscopy Australia as Research Data Manager. He works with microscopy facilities across the country to assist them in data management and develop good practices. He is the national lead for the work package on big-data microscopy in the Australian Characterisation Commons at Scale project.