High Performance Computing and Machine Learning – The Challenges of Supporting the Emerging ML Community with HPC Infrastructure
Miss Kiowa Scott-Hurley1, Dr Chris Hines1
1Monash eResearch Centre, Mooroolbark, Australia
Many research domains are beginning to leverage data science and neural network techniques. As research groups adopt these techniques they inevitably find that they need more computing power than their average laptop possesses and look to shared facilities such as high performance computing (HPC) clusters. HPC facilities offer large amounts of storage, large numbers of GPUs, and best of all – access is often paid for by the university rather than out of a research grant!
On the other hand, HPC facilities and their support procedures are often optimised for users who have improved their software, data access patterns and CPU usage over decades of development. Such facilities expect to have resources in use 24×7 in order to optimise the research return on a significant financial investment. In order to do so they expect users to queue their enormous computing jobs and come back when they are finished.
When novice data scientists using techniques optimised for instant iteration, using algorithms developed on local file storage, meet facilities which are optimised for overall throughput using large high bandwidth file storage, problems are bound to ensue.
In this BoF, we’d like to bring HPC professionals together to identify the common challenges of supporting ML, gather tools and resources which already exist to address these challenges, kickstart brainstorming on how to solve challenges without existing solutions, and form a community of HPC professionals looking to support the ML community moving forward, to overcome ML specific expertise barriers.
Biography:
Chris has been kicking around Australian eResearch for more years than he can remember. Chris exhibits the typical arrogance of physicists
assuming your problems are trivial, if you simply approximated your cows as spheres. An itinerant sys-admin, programmer and HPC consultant, he uses his skills wherever the needs of Monash university research require.
Kiowa is a pure mathematician at heart, and would never suggest the cow to sphere approximation is trivial! Her passion is supporting research with infrastructure and technology, bridging the HPC accessibility gap with improved technical documentation and training materials for researchers applying Machine Learning and AI techniques.