Benchmarking LLM Serving Systems for Agentic AI Workloads with XPerf

Computer Science > Distributed, Parallel, and Cluster Computing
[Submitted on 19 Jun 2026]
Title:Benchmarking LLM Serving Systems for Agentic AI Workloads with XPerf
View PDF HTML (experimental)Abstract:We present XPerf, a benchmarking framework that load-tests LLM serving systems with diverse agentic AI workloads. It provides detailed profiling of the serving system and hardware, enabling users to identify performance bottlenecks introduced by agentic workloads. Benchmarking LLM serving systems under agentic workloads is challenging - agentic applications rely on nondeterministic LLM outputs to guide their control flow; therefore, workload patterns vary unpredictably from run to run. XPerf minimizes this workload variation with a fine-grained trace replay approach: it enables users to easily collect traces from real agentic applications, synthesize new workloads with various patterns if needed, and reproducibly replay them on different LLM serving systems. XPerf includes eight agentic applications across diverse use cases (e.g., coding, deep research, and Q&A) by default. Our empirical study using these workloads shows that XPerf accurately replays agentic workloads, provides detailed performance breakdowns, scales to larger serving systems, and assists in serving system debugging. We will open-source XPerf on GitHub.
References & Citations
Loading...
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Connected Papers (What is Connected Papers?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
alphaXiv (What is alphaXiv?)
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Hugging Face (What is Huggingface?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Facts Only

* XPerf is a benchmarking framework for load-testing LLM serving systems with agentic AI workloads.
* The framework provides detailed profiling of the serving system and hardware.
* Agentic applications rely on nondeterministic LLM outputs to guide control flow, causing variable workload patterns.
* XPerf uses a fine-grained trace replay approach to minimize workload variation.
* Users can collect traces from real agentic applications using XPerf.
* Users can synthesize new workloads with various patterns.
* Workloads can be reproducibly replayed on different LLM serving systems.
* XPerf includes eight default agentic applications across use cases like coding, deep research, and Q&A.
* Empirical study shows XPerf accurately replays agentic workloads.
* The study shows XPerf provides detailed performance breakdowns and scales to larger serving systems.
* XPerf assists in serving system debugging.

Executive Summary

A benchmarking framework named XPerf is introduced for load-testing LLM serving systems under diverse agentic AI workloads. The framework aims to profile the serving system and hardware to identify performance bottlenecks introduced by agentic tasks. A core challenge addressed is the nondeterministic nature of agentic applications, which leads to unpredictable workload patterns between runs. XPerf mitigates this variability through a fine-grained trace replay approach, allowing users to collect traces from real agentic applications, synthesize new workload patterns, and reproducibly replay them across different LLM serving systems. The framework includes eight default agentic applications spanning use cases like coding, deep research, and Q&A. Empirical testing using these workloads demonstrates that XPerf accurately replays the agentic workflows, provides detailed performance breakdowns, scales to larger serving systems, and aids in debugging serving systems.

Full Take

The development of XPerf addresses a critical gap in evaluating LLM serving infrastructure: the performance impact of dynamic, non-deterministic agentic workloads. The necessity for trace replay suggests that traditional benchmarking methods fail when applied to systems interacting with generative models whose outputs govern execution flow, moving beyond static input/output testing into dynamic system behavior measurement. The value proposition lies in transforming a chaotic testing environment—caused by LLM outputs—into a controlled, reproducible experimental space. This moves the focus from measuring simple throughput to measuring the latency and resource consumption introduced specifically by agentic reasoning loops and subsequent control decisions. The architecture's emphasis on trace fidelity implies that performance bottlenecks are not just inherent to the hardware or serving software but are emergent properties of the complex interaction between the LLM, the orchestration logic, and the serving environment. Future work should explore how this replay capability can be used not only for debugging but also for designing LLM systems where agentic reasoning is intentionally optimized, perhaps by reverse-engineering optimal control flow patterns rather than merely mitigating performance degradation from unpredictable outputs. The assumption underlying this work is that if the workload can be accurately captured and replayed, optimization becomes possible across the entire agentic stack.