BEGIN:VCALENDAR
VERSION:2.0
METHOD:PUBLISH
CALSCALE:GREGORIAN
PRODID:-//WordPress - MECv6.5.6//EN
X-ORIGINAL-URL:https://conference.eresearch.edu.au/
X-WR-CALNAME:
X-WR-CALDESC:eResearch Australasia Conference
REFRESH-INTERVAL;VALUE=DURATION:PT1H
X-PUBLISHED-TTL:PT1H
X-MS-OLK-FORCEINSPECTOROPEN:TRUE
BEGIN:VEVENT
CLASS:PUBLIC
UID:MEC-c71c14199fd7d86b0be2a0d4ee4c738f@conference.eresearch.edu.au
DTSTART:20211014T045000Z
DTEND:20211014T051000Z
DTSTAMP:20210824T192300Z
CREATED:20210825
LAST-MODIFIED:20210906
PRIORITY:5
TRANSP:OPAQUE
SUMMARY:Workflow Orchestration on HPC with Airflow, Cookie-cutter Templates and Jupyter Notebooks
DESCRIPTION:Dr Blake Seers1, Ms Claire Trenham2, Dr Ron Hoeke1, Dr Paul Branson3\n1Csiro, Aspendale, Australia\n2Csiro, Black Mountain, Australia\n3UWA, Crawley, Australia\nWorkflow orchestration is essential for running large numerical models on high performance computers (HPCs). Airflow is a python workflow orchestration tool created by AirBnB that uses directed acyclic graphs (DAGs) to build up a workflow based on the relationships between individual tasks. We use Jupyter notebooks and our own open-source python library to interactively build the DAG. Once we’ve finished developing the DAG we can then trigger it inside the Airflow web server’s user interface. Cookie-cutter templating is used to populate the model directory with the necessary input files and parameters to run the model. The directory is then moved onto the HPC where the model is triggered using an SSH command. I will cover all these steps and more in this presentation to demonstrate how we have developed our own DAGs to schedule and run large numerical models on HPCs.\n\nBiography:\nBlake joined CSIRO’s Ocean & Atmosphere in 2019, working within the Sea Level, Waves and Coastal Extremes team. Before joining CSIRO, Blake completed a PhD in Marine Science and Statistics, and worked a statistical consultant within the Department of Statistics at the University of Auckland. Blake works across various projects within the team where he contributes to the team’s growing codebase and scientific computing requirements.\n
URL:https://conference.eresearch.edu.au/events/workflow-orchestration-on-hpc-with-airflow-cookie-cutter-templates-and-jupyter-notebooks/
END:VEVENT
END:VCALENDAR