Sun C1, Guo S1
1Sydney Informatics Hub, Core Research Facilities, The University Of Sydney, Camperdown, Australia
Biography:
Dr Chao Sun is the Data Science Group Lead at the Sydney Informatics Hub, Core Research Facilities, University of Sydney, overseeing two development teams that support national research platform projects. With a background in electronics engineering and machine learning (PhD), Chao has spent over a decade collaborating with domain experts across disciplines. Chao began in the Faculty of Arts and Social Sciences, supporting humanities and social science researchers, then expanded to more faculties after joining the Sydney Informatics Hub in 2019. Since 2022, Chao has led development of the text analytics suite for the Language Data Commons of Australia (LDaCA), an ARDC–dHASS partnership, and since 2024 has contributed to the Australian Plant Phenomics Network (APPN).
Abstract:
Qualitative researchers are already using generative AI, mostly through chat interfaces that are not designed for research. In supporting them, we keep hitting the same three failures. Results are irreproducible: the same question yields different answers, with no record of prompt or model. Costs are high and unverifiable: in one real case, a researcher uploaded a gigabyte-scale dataset to a chat service and paid more than twenty dollars for questions like "when did this phrase first appear? said by whom, in what context, and how did others respond?", with no way to validate a single answer. And responses arrive as unstructured text that cannot feed counting, statistics, or any further analysis.
We are building AI Annotator, a module of Wordflow, the Language Data Commons of Australia's open-source text analytics platform, that treats these as design constraints rather than user error. We start from a simple premise: AI should not replace critical thinking in research, and no model deserves blind trust. The module supports the whole qualitative workflow: researchers explore their data, form a coding scheme, annotate manually, and verify inter-coder reliability at a scale they control. Only after the scheme is validated does the AI help scale it up to the larger corpus: it is guided by the researchers' own annotated cases, reports confidence alongside its results, and remains open to the same reliability checks afterwards. Each run is a controlled operation: a prompt template is applied document by document, and responses are constrained into structured columns beside the source text – every annotation traceable, auditable against the original wording, countable, and reusable by other analysis tools. The prompts, model and parameters are part of the workflow, making the run reproducible. The tool also supports local AI endpoints, so it is potentially suitable for processing confidential text data.
