Scalable bioinformatics workflows for growing ‘-omics’ datasets on National Collaborative Research Infrastructure Strategy Facilities
Dr. Tracy Chew1,2, Georgina Samaha1,2, Dr. Ove J. R. Gustafsson1,3, Dr. Sarah Beecroft1,4, Dr. Marco De La Pierre1,4, Dr. Nigel Ward1,5, Dr. Rosemarie Sadsad1,2
1Australian BioCommons, , Australia
2Sydney Informatics Hub, University of Sydney, Australia
3Melbourne Bioinformatics, University of Melbourne, Australia
4Pawsey Supercomputing Centre, Australia
5Queensland Cyber Infrastructure Foundation, Australia
Australian researchers rely on High Performance Computing (HPC) to process increasingly large life science and ‘-omics’ datasets. Bioinformatics workflows are often complex and, unlike traditional HPC, orchestrate multiple data, memory, I/O, and/or time-intensive compute tasks that mismatch local infrastructure paradigms. As a part of the Australian BioCommons, we have re-engineered popular bioinformatics workflows to enable scalable and efficient use of National Collaborative Research Infrastructure Strategy (NCRIS) eResearch facilities.
Re-engineered workflows and end-user resources were developed to meet FAIR data principles. Workflows identified through national surveys were optimised in collaboration with multiple research groups and national compute facility specialists. This ensured user-friendliness, applicability across research interests, datasets, and optimal use of local HPC architecture. Workflows are registered on WorkflowHub for findability and pipeline adoption is supported through training, user guides and documentation that follow FAIR recommendations by the Australian BioCommons.
Developed workflows are available through the Australian BioCommons WorkflowHub space (https://workflowhub.eu/programmes/8). Between 2019-21, optimised workflows were successfully adopted by independent groups across pathogen, human, domestic animal, agricultural, and wildlife research. They have contributed to successful Australian competitive grants, national/local HPC allocation applications and research publications.
Creating efficient and scalable bioinformatics workflows on Australian compute infrastructure requires custom development to meld specific infrastructure hardware, usage and access policies with domain best practices. We are refining support and ongoing maintenance models and aim to enhance existing pipelines by adding auto-deployment capabilities and portability to other compute infrastructure accessible to Australian researchers.
Tracy enjoys developing fast, scalable big data analytics workflows on cloud and national compute facilities. She has a background in animal health, genomics, transcriptomics and bioinformatics having completed her Bachelor’s degree in Animal and Veterinary Bioscience (Hons I, USyd) and her PhD (USyd) on canine de novo mutations and Mendelian disorders. Since joining the Sydney Informatics Hub in 2017 and the Australian BioCommons in 2020, she has been involved in domestic animal genomics, human health (microbiome, influenza, sepsis, pancreatitis, cancer research), metagenomics, plant ‘omics (food security), Australian wildlife and conservation research.