UQ’s Long Term Storage Journey: Co-design, value, risk and balance.
Jake Carroll1 1The University Of Queensland, Forest Lake, Queensland, 4078
Abstract
In 2019, the University of Queensland needed to decide on what its long term research data storage strategy would look like for the next several years. Separate from but integrated with UQ’s now successful research data storage fabric strategy, this project was about the long term preservation layer.
With an ageing platform that had been in place for around ten years (HPE’s DMF6), approaching scalability limits, UQ had to make architecture, design and strategy decisions that would carry the organisation for another ten years or more.
UQ eventually chose to implement HPE’s DMF7. This choice was informed by an in-place migration path rather than a manual copy, that a co-design possibility existed between UQ and HPE, and that, in UQ’s view, the natural end of POSIX based filesystem representations for mass archive tasks was close.
The task ahead of UQ and HPE was to migrate ~4.8 billion objects over the Christmas and New Years eve period rolling into 2020. Once complete, UQ and HPE needed to find a way to migrate out of legacy tape and filesystem storage technologies in an expedient way, to mitigate further risks.
The outputs of these tasks was a system that UQ and HPE co-designed in order to meet requirements, creating value for researchers. This presentation details the complexities, challenges, the value proposition and the strategy of risk management in order to balance cutting edge product and design with the most critical data assets of an organisation.
Biography
Jake Carroll is the current Chief Technology Officer of The University of Queensland’s Research Computing Centre. Jake is an internationally recognised expert in the design, governance, management and procurement of large research data storage and supercomputing infrastructure and the governance of associated facilities. He has consultancy experience in various sectors including health, defence, government, research and private industry.