PITSCHI: Particle Imaging depoT using Storage CacHing Infrastructure
Mark Endrei1, Nishanthi Dasanayaka1, David Abramson1, Rubbiya Ali2, Tom Mason2, Roger Wepf2 1Research Computing Centre, The University Of Queensland Australia2Centre For Microscopy And Microanalysis, The University Of Queensland Australia
Abstract
Scientific imaging instruments with modern fast CMOS detectors are generating increasingly large datasets, and as a result, data management becomes more critical. This is particularly true in large multiuser facilities such as the Centre for Microscopy and Microanalysis (CMM) at UQ as it operates a wide range of microscopes and many of them are big data producers. A central data repository to store, index, and annotate data not only allows its researchers to search, browse and retrieve their data easily but also harvest metadata to enrich these datasets.
Pitschi is a central data repository for CMM instruments that adheres to FAIR data principles. It was developed within the Australian Characterisation Commons at Scale project, funded by the Australian Research Data Commons. Pitschi is based on the opensource Clowder data management framework, and fully integrates with the instrument booking system and storage infrastructure at UQ. Pitschi provides an end-to-end data management process, from capturing raw data to transferring them to storage collection, ingesting/indexing the data into the repository, and automatically extracting metadata of supported file types. These metadata are used to facilitate search and discovery. The data ingested in Pitschi are available in various platforms such as HPCs, personal computers, and processing platforms such as CVL. Data transport is arranged transparently using the Metropolitan Data Caching Infrastructure (MeDICI).
We presented Pitschi at eResearch Australasia 2021 and 2022. This year we aim to share our experience rolling out Pitschi to a wide range of imaging researchers and instruments at UQ.
Biography
Dr Mark Endrei completed his PhD with the School of Information Technology and Electrical Engineering at the University of Queensland, Australia. He currently works at the Research Computing Centre as a Senior Research Software Developer helping UQ researchers take advantage of range of high performance computing services. He also has more than 20 years of experience in IT industry, working with large corporations both nationally and internationally.