AgentX Benchmark Datasets

How an AgentX run is built

1. 01

   Capture

   An opt-in HTTP proxy records request and response timing, token counts, conversation IDs, and subagent IDs as sessions run.

2. 02

   Transform

   Original prompts, source code, tool arguments, and tool results are removed. Inputs become session-scoped chained hashes in 64-token blocks, preserving matching prefixes without revealing content.

3. 03

   Reconstruct

   AIPerf fills those blocks with deterministic synthetic coding and tool-use tokens, then rebuilds each session as a directed acyclic graph (DAG) of main-agent turns, parallel subagents, auxiliary requests, and inter-turn tool time.

4. 04

   Replay and measure

   A seeded warmup establishes cache state before each configuration is profiled for one hour across a sweep of concurrent clients. Per-replay cache-bust markers stop unrelated sessions from sharing prefixes.

What is in the v1.0 dataset

AgentX v1.0 uses 393 Claude Code sessions selected from an internal, opt-in trace corpus. Eligible sessions contain at least 20 requests, use Claude Code 2.1.139 or newer, and have no more than 10 concurrent subagents. Processing removes duplicate requests, client-specific security-monitor and title-generation calls, and reconstructed inputs above 990k tokens.

• full: The complete AgentX v1.0 replay set, including contexts up to 1M tokens.
• 256k: A context-limited variant for models and inference engines configured with a maximum context of 256k tokens.

Replay controls

Steady-state start

A fixed seed selects a point 25–75% through each conversation. Primer requests and 10 additional warmup requests per replay lane materialize KV cache before profiling begins.

Deterministic replay

The seed fixes conversation sampling, starting points, and synthetic content. Reported metrics cover only the one-hour profiling window, not warmup.

Speculative decoding

Because synthetic tokens can distort draft-token acceptance, AgentX forces an acceptance length measured with SPEED-Bench for each model, speculator, draft length, and thinking-mode combination.

DRAM offload

Servers without standardized DRAM are capped at 3 TB. Standard GB200 NVL72, GB300 NVL72, and TPUv7 systems use installed capacity, and every configuration may access only the share proportional to its GPU allocation.

How to read AgentX results

Concurrency means concurrent agent clients, not a fixed request batch. AgentX is closed loop, so faster configurations complete more requests and can encounter a slightly different workload mix. This variation is most visible at low concurrency. Report throughput with time to first token (TTFT) and interactivity; a single latency statistic does not describe the full run.

AgentX measures serving-system performance. Its synthetic payloads do not support model-quality evaluation.

The client cannot observe provider-side chat templates, proprietary tokenizers, server tools, encrypted reasoning content, or the exact token expansion of images and documents. AgentX uses deterministic placeholders and model-specific padding for those inputs. The resulting traces reproduce request lengths, timing, conversation topology, and KV-reuse patterns. They do not contain the original conversations.