Nvidia GB300 NVL72 cuts AI inference costs by up to 7.2x

Tests pairing vLLM and Inferact use 72 copper-linked GPUs to run MoE models faster while cutting inference costs.

Semiconductor research firm SemiAnalysis released benchmark figures for the Nvidia GB300 NVL72 rack running artificial intelligence inference workloads. Developed through co-design among the open-source vLLM project, inference startup Inferact, and Nvidia engineers, the system achieved up to 7.2 times higher performance per total cost of ownership (TCO) than competing setups when serving Mixture-of-Experts (MoE) architectures.

The primary measurements come from runs on the DeepSeek v4.1 Flash model formatted in FP4 precision. On the public InferenceX telemetry dashboard, the rack delivered a steady throughput of 77,931 tokens per second per GPU at a service target of 50 tokens per second per user. Assuming an hourly cost of $2.31 per GPU at major cloud providers, the calculated operational expense landed at roughly $0.008 per million generated tokens.

Inside the GB300 NVL72 unified rack

The efficiency gain did not come from raw clock speeds on isolated accelerators. Instead, it stems from how the hardware moves weights across processors. Standard AI servers pack eight GPUs connected by high-speed printed circuits. As soon as a model outgrows that single chassis, communication falls back to external network cables that max out around 50 GB/s per chip, creating persistent transfer bottlenecks.

In the GB300 NVL72, Nvidia grouped 72 Blackwell Ultra GPUs into one continuous physical domain connected by a copper backplane without optical transceivers. The NVLink 5 interconnect pushes 900 GB/s per chip in each direction. That massive bandwidth allows what engineers call wide expert parallelism (wideEP). Parameter sets for each MoE expert are distributed across all 72 chips. Because each GPU holds only a small fraction of the model weights, local memory pressure drops and the system avoids shuffling duplicate data across nodes for every query.

HBM3e capacity and the GB200 comparison

Memory sizing also altered the cost equation. The GB300 features 288 GB of HBM3e per GPU, up from 192 GB on the prior GB200 generation. When testing the 1.6-trillion-parameter DeepSeek-V4-Pro model, that added memory headroom let operators run larger batch sizes during the prompt prefill stage without choking the subsequent token decode phase. Measurements recorded on InferenceX showed 2.83 times higher token throughput per GPU and a 2.31-fold improvement in cost-to-performance over the GB200 NVL72.

Test conditions and unverified projections

These benchmark results come with specific operational boundaries. The tests ran on early-access machines hosted by CoreWeave, with direct optimization support from Nvidia and Inferact engineers who tuned custom software kernels for the setup. Hyperscale operators like AWS and Google Cloud have not yet published comparable data from standard multi-tenant production clusters.

SemiAnalysis also projected that sharper latency targets, between 100 and 150 tokens per second per user, could lift performance advantages to 10x or 20x. [RUMOR]: That broader projection remains an internal extrapolation from the research firm and is not backed by validated data on the public InferenceX dashboard.

What shifts for AI data center infrastructure

For engineering teams sizing data centers for sparse architectures, these benchmarks highlight a clear shift in hardware design. Standalone servers with eight GPUs struggle to sustain the high-frequency weight transfers that MoE architectures demand. Real economic gains now depend on treating the entire 72-chip rack as a unified computer and aligning open-source runtime engines to take advantage of direct copper fabric.

Sources

• SemiAnalysis announcement on X: https://x.com/SemiAnalysis_/status/2110496505031192895
• Technical architecture breakdown by SemiAnalysis: https://x.com/SemiAnalysis_/status/2110496506578890963
• Latency projections on X: https://x.com/SemiAnalysis_/status/2110496508285960571
• vLLM engineering announcement: https://x.com/vllm_project/status/2092040745842774377
• DeepSeek v4.1 Flash telemetry on InferenceX: https://inferencex.semianalysis.com/run/deepseek-v41-flash-on-gb300-nvl72
• GB300 vs GB200 comparison on InferenceX: https://inferencex.semianalysis.com/blog/gb300-nvl72-vs-gb200-nvl72-dsv4-pro-vllm-fp4
• Nvidia GB300 NVL72 technical resources: https://resources.nvidia.com/en-us-inference-contact-us/discover-ai-inference