Making Terra-Bytes of data accessible in ‘web-time’!

Mr Uwe Rosebrock1, Mr Simon Pigot1

1Csiro, Hobart, Australia, uwe.rosebrock@csiro.au

 

ABSTRACT

The  Australian  Wave  Energy  Atlas  (AWavEA)  portal  provides  access  to  a  32-year  hind-cast  of  wave  data  from  the Australian region at an hourly temporal resolution. In its entirety, it consists of nearly 20 TB of data of which a subset – 5 TB, is used to provide real-time time-series  analysis. In average a web-user’s  expectations  to wait for a asynchronous query is in the order of 10’s of seconds. With some simple measures the supporting data was prepared to allow analysis processes to return results covering over 300,000 records of data in under 10 sec. This is an improvement of 3 magnitudes to the standard layout of the data.

With increasing amount of data available and the cross-disciplinary  use, it is no longer feasible to simply copy or query large data holding remotely. Query processes are necessary in front of data, an example is the NCI data-cube. We like to present simple measure which allow improved access and make incorporation in spatial portals feasible.


Biography:

Uwe Rosebrock is a Senior Software Engineer at CSIRO Oceans and Atmosphere in Hobart. He has extensive experience in large data processing, software design as well as project and defect management. Uwe leads a team of software engineers, who led the development of the ARENA-CSIRO Australian Wave Energy Atlas, and its integration into AREMI, and also developed CSIRO’s relocatable modelling system and the DIVE visualisation packages as part of the CSIRO/BoM/Navy BlueLINK program.

About the conference

eResearch Australasia provides opportunities for delegates to engage, connect, and share their ideas and exemplars concerning new information centric research capabilities, and how information and communication technologies help researchers to collaborate, collect, manage, share, process, analyse, store, find, understand and re-use information.

Conference Managers

Please contact the team at Conference Design with any questions regarding the conference.

© 2016 - 2017 Conference Design Pty Ltd