Fox C1
1Weka, , Australia
Biography:
Callan Fox is a leading voice on the economics and architecture of AI inference infrastructure. As the product lead behind WEKA’s Augmented Memory Grid, he focuses on overcoming the "GPU memory wall" to optimize long-context, agentic workloads, extending usable memory capacity and drastically reducing time-to-first-token. His work centers on the thesis that the next era of AI infrastructure will be defined not by peak compute, but by data mobility and token-serving efficiency.
A hands-on engineer and strategist, Callan regularly publishes technical benchmarks on KV-cache optimization and distributed inference performance and has championed open-sourcing core benchmarking tools for the broader AI development community. He has featured as a speaker at major industry stages including SC25, the AI Engineer Summit, and MTX 2026. Prior to his work at WEKA, Callan spent over a decade designing and building high-performance, mission-critical data and storage systems at enterprise scale, including managing core platform portfolios at Dell Technologies and EMC.
Abstract:
Artificial intelligence is rapidly becoming a foundational tool for research across science, engineering, medicine, and the humanities. While most infrastructure planning has focused on model training and traditional high-performance computing (HPC), a new class of workloads is emerging: agentic AI. Unlike conventional inference, agentic systems perform iterative reasoning, invoke multiple models and tools, retrieve large volumes of contextual data, and execute complex workflows on behalf of researchers.
These characteristics create infrastructure demands that differ significantly from both HPC simulations and AI training environments. Traditional performance metrics such as FLOPS, training throughput, and GPU utilization provide only a partial view of system effectiveness for agentic workloads.
This paper presents an opensource framework for benchmarking agentic AI workloads in research environments. We examine workload characteristics including context retrieval, memory utilization, concurrency, latency sensitivity, and end-to-end task completion. Through representative agentic workflows, we identify emerging bottlenecks across compute, memory, storage, and networking infrastructure, and explore how these factors impact researcher productivity and system efficiency.
The framework is informed by testing and operational observations from large-scale AI cloud and service-provider environments, including deployments supporting research and enterprise AI workloads at CoreWeave, Oracle Cloud Infrastructure (OCI), Nebius, and Firmus Technologies. These examples provide practical insight into how agentic workloads behave at scale and where conventional infrastructure assumptions begin to break down.
The paper concludes with recommendations for research institutions evaluating infrastructure investments to support the next generation of AI-enabled discovery. As agentic AI moves from experimentation into production research workflows, understanding these new performance dynamics will become increasingly important for future eResearch platforms.
Github for current tools: https://github.com/callanjfox/kv-cache-tester
