Nvidia has released benchmark results showing that its GB300 NVL72 can support 61,400 concurrent AI agents per megawatt in the new AA-AgentPerf benchmark published by independent evaluator Artificial Analysis. For comparison, the previous-generation H200 posted roughly 2,600 agents per megawatt, leaving the GB300 with an advantage of about 20x. The result stands out not just because of the hardware delta, but because the benchmark itself uses a different measuring stick from traditional inference tests.
According to Nvidia, the GB300 NVL72 delivered around a 20x lead over H200 under both the 20 tokens-per-second and 60 tokens-per-second service tiers defined in AA-AgentPerf. Rather than focusing on raw token throughput, the benchmark centers on “agents per megawatt,” a metric intended to capture how many AI agents a system can sustain under a fixed power budget while still meeting service targets.
Why the industry is moving beyond token throughput
Artificial Analysis describes AA-AgentPerf as the first inference benchmark built specifically for AI agents. That distinction matters because most existing inference benchmarks were designed around fixed-length, synthetic requests. Those tests generally measure how fast a system handles one prompt-response exchange or how many such requests it can process concurrently, but they are not designed to reflect the behavior of agentic workloads.
Agent workloads are structurally different from ordinary chat interactions. A normal conversational request may resemble a sprint: one prompt goes in, one answer comes out, and the transaction ends. An agent task is closer to a relay. It can break a goal into dozens or even hundreds of steps, repeatedly calling a model, reading files, writing code, running commands, checking outputs, invoking tools, and deciding what to do next based on intermediate results.
As those steps accumulate, the context window grows, tool outputs get appended, and the overall workload becomes much more complex than a single inference pass. In that environment, long-session behavior, KV cache reuse, scheduling efficiency, memory hierarchy, and the handling of mixed long and short phases can materially change performance. Traditional synthetic benchmarks often fail to capture those system-level stresses.
That is why the new benchmark replaces “tokens per second” as the main headline metric with “concurrent agents per megawatt.” For operators buying GPUs or planning data center deployments, the practical question is not just how many tokens a model can emit, but how many useful agent sessions a given power and hardware budget can keep alive.
How AA-AgentPerf is constructed
Instead of feeding models fixed synthetic prompts, AA-AgentPerf replays real coding-agent trajectories. These trajectories were gathered from agents solving issues in real code repositories, spanning more than 12 programming languages. A single session can run for as many as 200 turns, and context length can exceed 100,000 tokens. Input lengths range from 5,000 to 130,000 tokens, with an average of about 27,000 tokens.
Importantly, the benchmark is not trying to maximize concurrency at any cost. It first defines service-level objectives, or SLOs, that constrain acceptable output speed and time-to-first-token. Only after those requirements are met does the benchmark measure how many concurrent agents the system can sustain. AA-AgentPerf includes multiple service tiers, from a usable 20 tokens per second to as high as 180 tokens per second, and records maximum concurrency separately for each tier.
The benchmark also allows optimizations that vendors actually use in production, such as KV cache reuse, speculative decoding, and disaggregated prefilling and decoding. Earlier benchmarks often disabled these optimizations to simplify comparison, but Artificial Analysis argues that doing so produces results with less operational relevance. AA-AgentPerf also tracks output quality so that systems cannot inflate concurrency simply by degrading responses.
GB300’s edge appears in both power efficiency and GPU density
In a test using what Nvidia described as a leading frontier-class mixture-of-experts model, the GB300 NVL72 achieved 61,400 concurrent agents per megawatt. On a per-GPU basis, the system averaged 57.5 agents per GPU. The H200 comparison point came in at about 2,600 agents per megawatt and 1.4 agents per GPU. That implies roughly a 20x advantage on the power-efficiency metric and about a 40x advantage in service density per GPU.
Those two numbers matter for different reasons. Agents per megawatt is fundamentally an energy-efficiency and capacity-planning metric. It tells buyers how much agent throughput a fixed power envelope can support. Agents per GPU, by contrast, is more directly tied to hardware utilization and service density. Together, they offer a more operationally useful view of system economics than token throughput alone.
Nvidia also noted that the first batch of benchmark results includes not only GB300 but also products such as AMD’s MI355X, across configurations ranging from single GPUs to servers and full rack-scale systems. The early pattern is that rack-scale systems tend to perform better because they can spread inference work more efficiently across larger pools of accelerators. The transition from Hopper to Blackwell also appears to represent a meaningful step change rather than an incremental improvement.
The result is fundamentally a rack-scale system story
From Nvidia’s perspective, the GB300 result is not just a single-chip win. A key factor is that the 72 GPUs in the GB300 NVL72 are linked into a high-bandwidth rack-scale system through NVLink. For large mixture-of-experts models, that matters because model components and experts can be distributed across many GPUs instead of being forced into narrower device boundaries.
Nvidia said the CUDA stack helps overlap expert communication with computation, reducing the visible latency cost of coordinating specialists across the system. TensorRT-LLM is used to maintain efficiency as concurrent sessions climb, including by optimizing prefilling and decoding as separate phases. In effect, the published benchmark score reflects a combination of silicon, interconnect, and software stack improvements rather than a raw chip-speed comparison in isolation.
With 72 GPUs functioning as a tightly coupled high-bandwidth pool, the system can share parameters, KV cache, and intermediate states more effectively. That systems-level integration is what makes a benchmark result on the order of 60,000 concurrent agent sessions possible. Without rack-scale interconnect and software coordination, gains in individual GPU capability alone would be less likely to translate into smooth large-scale agent serving.
Important caveats around the benchmark
There are also limits to how these figures should be interpreted. First, 61,400 concurrent agents does not mean one machine is simultaneously running 61,400 fully independent large-model instances. Under the benchmark design, the system is serving simulated concurrent sessions based on prerecorded trajectories. Tool calls are not truly executed; instead, they are represented by fixed CPU-time delays so the benchmark can isolate inference-system differences as much as possible.
Second, benchmark scores are not the same as production service guarantees. Artificial Analysis has described AA-AgentPerf as a rapidly evolving snapshot of a frontier market, and it has indicated that many systems may still have headroom left to unlock through software improvements. That means the benchmark is better understood as a comparative framework for system design and efficiency trends than as a direct proxy for real-world service-level agreements.
Third, AA-AgentPerf is still a benchmark proposed by a single organization. It remains unclear whether it will eventually achieve the industry-wide standing of MLPerf. Even so, the benchmark’s design points to a broader shift already underway: as AI agents become a more important workload, evaluation is moving from model-centric throughput metrics toward system-level capacity, service density, and the number of agents that can be sustained per unit of power.