Transforming Research Code to A Modern Web Architecture – Pipetools

Paulus Lahur, Kieran Lomas

CSIRO, Clayton, Australia,

CSIRO, Clayton, Australia,



In this paper, we outline the process of transforming research code to a web application, using Pipetools project as the study case. The target is to reach a wide audience of users who can benefit from the code. We are constructing infrastructure and code that support and encapsulate the research code to significantly improve its usability, as well as expand its modes of usage. Currently the project is moving along at reasonable pace. We would like to share challenges that we face and the thought process in solving them. The lessons learned here will hopefully benefit researchers and software developers working in similar projects.


Research code is a highly valuable asset hidden deep inside research institutions. It typically runs on a very specific device and environment, and is accessible only to a few researchers. Throughout the course of the research, algorithms and codes are accumulated and improved. As the research matures, the potential benefit to other people increases. In many cases, there will be people out there who are willing to give money to be able to use the software. The problem is, the software is practically usable only to those who makes it, or at least to those who have intimate understanding of how it works. In order to make the software usable to a wider audience, another stage of software development is required. More code needs to be built around the research code in order to improve user experience.

There are two major approaches here. The first one is to make “native application,” that is, software that is native to a certain operating system. In fact, the research software itself belongs to this type. The other approach is to turn it into web application, that is, software that runs on remote machine. Although there are many pro s and cons for either approach, we opt for the latter, because it is accessible to people on various operating systems, and is therefore easier to support and maintain. Software licensing and protection becomes simpler. Rolling out a new version is also trivial. Furthermore, web application also opens door to collaborative work, where a number of people, possibly on different parts of the World, are working on the same set of data.


In order to develop an effective solution, we need to create a modular system, where developers can focus on specific modules. This is outlined in Figure 1. In essence, the development is split into these parts:

  • Front End. It deals with user interface. It translates user commands and send them to Back End.
  • Back End. It receives commands from Front End and calls the research code to do the actual computation.
  • Infrastructure. It deals with services that enable Front and Back Ends to work. This includes containers, as well as continuous integration and deployment.

Each parts have their own challenges. Details of the system will be presented in the paper.

Figure 1: Simplified layout


Research code: Lionel Pullum (Mineral Resources, Clayton VIC) Project manager: Andrew Chryss (Mineral Resources, Clayton VIC) Team lead on IMT side: Daniel Collins (IMT, Kensington WA)

Front End: Kieran Lomas (IMT, Clayton VIC)

Back End: Paulus Lahur, Sam Moskwa (IMT, Clayton VIC)

Infrastructure: Dylan Graham, Andrew Spiers, Sam Moskwa (IMT, Clayton VIC)

Paulus Lahur is CSIRO staff since 2015. He is in Scientific Computing of IMT.

Recent Comments