Extracting and Analysing Twitter and Facebook Data: Development of an User-Friendly Platform

Dr Yoong Kuan Goh1, Dr Jerry Lai1, Dr Christopher McAvaney2

1Intersect Australia, Sydney, Australia
2Deakin University, Melbourne, Australia

Social media platforms, such as Twitter and Facebook, have in recent years become the primary space whereby influencers can share their thoughts, ideas and feelings about different topics, and over the past few months, the global COVID-19 pandemic. By examining the data from these platforms, researchers are able to discover patterns and interesting insights into real-world problems. The biggest challenge that many researchers encounter when conducting Twitter and Facebook research – How do I extract data from these platforms without having to manually scrape data page-by-page, post-by-post? In addition to the cost consideration, another problem with social media research is the difficulty associated with cleaning, transforming, filtering and analysing the data.

Solution: Using an open source Twitter API, Deakin eResearch and Intersect Australia are developing a user-friendly tool to help researchers collect, clean, transform, filter and analyze online social data from Twitter and Facebook. Since the availability of public social media data has also presented ethical challenges, we will consider the ethical aspect too when designing the tool.

Methodology/Development: We began the development process by gathering requirements from a number of researchers specialised in the discipline of journalism, social science, and digital humanities. With this information, we are developing the user-friendly platform in Python, which focuses on three stages that are data collection, data manipulation and data analysis.


Yoong (Andrew) has a strong background in computer science and mathematics. His work interests are natural language processing, data analysis and machine learning.

Jerry has a background in psychological science and statistics. Jerry works with multiple research teams in Deakin on survey design; statistical programming; data-analytics and visualisation and promoting statistical literacy.

Christopher McAvaney is Services Manager, eResearch at Deakin University. He is responsible for establishing an eResearch program of work at Deakin which includes REDCap.  Christopher has extensive research administration experience at Deakin and is keen to leverage as much as possible from REDCap via its extensible design.

Aggregating Cultural Heritage Data to Interrogate the History of Colonial Collecting: Reconstructing Western Australian Collections in the “Collecting the West” Project

Dr Toby Burrows1

1School of Humanities, University Of Western Australia, Nedlands, Australia


The “Collecting the West” project has been identifying indigenous objects and natural history specimens which were originally collected in Western Australia and are now located in institutions across Great Britain, Europe, and North America, as well as eastern Australia. These materials were collected by Western explorers, colonial administrators, and entrepreneurs.


Assembling the descriptive data and digital images for these items into a single database is essential for giving an overall picture of what was collected and when, and for documenting who the collectors were, and which museum and library collections acquired these materials. The “Collecting the West” database is using the nodegoat software, which provides data management, network analysis, and a visualisation environment designed specifically for historical humanities data.


The presentation will discuss and demonstrate the different ways in which the data can be explored and visualized. It will also consider issues which have arisen during the database construction, especially those related to data modelling and linking across a complex range of heterogeneous sources. The database’s potential for addressing broader questions will also be discussed: how these materials ended up in their current locations, why specific museums and libraries acquired them, and why they were collected in the first place.


Finally, the presentation will consider the extent to which such an aggregation of data can contribute to current and future understandings of the history and cultural heritage of Western Australia, and its relevance for digital repatriation and restitution of indigenous objects.


Toby Burrows is a Senior Honorary Research Fellow in the School of Humanities at the University of Western Australia and a Senior Researcher in the Oxford e-Research Centre at the University of Oxford.

His current projects include “Collecting the West”, funded by the Australian Research Council (2016-21), and “Mapping Manuscript Migrations”, funded by the Trans-Atlantic Platform under its “Digging into Data Challenge” (2017-20).

His recent publications focus on the use and reuse of collections data from cultural heritage institutions, the history of cultural heritage collections, and the history of medieval manuscript collecting.

Recent Comments


    About the conference

    eResearch Australasia provides opportunities for delegates to engage, connect, and share their ideas and exemplars concerning new information centric research capabilities, and how information and communication technologies help researchers to collaborate, collect, manage, share, process, analyse, store, find, understand and re-use information.

    Conference Managers

    Please contact the team at Conference Design with any questions regarding the conference.

    © 2018 - 2020 Conference Design Pty Ltd