Dr Ahmed Shamsul Arefin1
1CSIRO, ,
Introduction
We present a simple approach for creating a portable HPC cluster environment with the aid of VirtualBox and Vagrant automation. The one-line deployed environment may act as an effective playground for the power users and sysadmins to eliminate some of the crucial HPC software stack incompatibilities at a much early stage.
Methods
We used the following four main software tools: VirtualBox, Vagrant with Ansible and an OS image: RHEL. The Vagrant is an open-source free software for building and maintaining portable virtual software development environments. It uses provisioners and providers as building blocks to manage the development environments. The “provisioners” are tools that allow users to customize the configuration of virtual environments such as “Ansible” and the “providers” are the services that Vagrant uses to set up and create virtual environments such as “Virtualbox”. We create simple a vagrant deployment that automatically creates one master and four compute VMs with varying memory and storages, attach the nodes to a private network and finally add some HPC software such as Slurm. The playbook files for each of these nodes are written in the YAML format and is stored in ~/portablehpc/playbooks/. Users may further customize these playbooks to upgrade the nodes with the latest packages.
Conclusion
The automated VM deployment process creates a playground for testing the HPC software stack. It is fully reproducible, re-programmable and may aid the finding of software incompatibility and issues at an early stage and hence, reduce end user frustrations and enhance the HPC experience.
Biography:
Dr Ahmed Arefin is a Computation Scientist working within the HPC Systems Team, Scientific Computing Platforms, CSIRO. He completed his PhD in Computer Science (Data-Parallel Computing & GPUs) from the University of Newcastle, Australia and worked as a Postdoctoral Researcher (Parallel Data Mining) at the Centre for Bioinformatics, Biomarker Discovery & Information-Based Medicine (CIBM), The University of Newcastle, Australia. His research interest focuses on the application of HPC in data mining, graphs and visualization