How NVIDIA Vera Rubin Gets More Inference from the Same HBM

What changes when HBM capacity stays flat

In its November 5 technical article, NVIDIA describes how Vera Rubin can improve inference efficiency while retaining 288 GB of HBM per GPU. On SemiAnalysis AgentX, which replays recorded coding agent sessions with tool call gaps and growing context, NVIDIA reports that Vera Rubin NVL72 delivers up to 30 times the throughput per megawatt of GB300 NVL72 at 160 tokens per second per user. This is a workload-specific system comparison with an explicit interactivity condition. NVIDIA Technical Blog

For operators, capacity planning must account for weights, active context, and the cost of bringing cached state back. Identical memory capacity can support different amounts of useful work when idle residency and repeated prefill change. Our interpretation is that reducing these sources of waste connects the five mechanisms. The supplied material does not isolate each component's contribution to the headline result, or establish the speed, cost, or required deployment size for an arbitrary model.

NVFP4 and expert placement free different bytes

NVFP4 encodes values in four-bit E2M1, with an FP8 E4M3 scale shared by each block of 16 values and an additional FP32 scale per tensor. NVIDIA reports weight-footprint compression ratios of approximately 3.5 relative to FP16 and 1.8 relative to FP8, and up to 50% less KV-cache space than FP8. Scaling metadata occupies memory, so nominal bit width alone cannot determine the full saving. Accuracy must still be evaluated for the chosen model and task. NVIDIA Technical Blog

TensorRT-LLM's wide expert parallelism distributes MoE experts across eight or more GPUs, reducing each GPU's share of expert weights and making room for cache and larger batches. Our inference is that arithmetic intensity improves when enough tokens reach an expert to reuse its loaded weights effectively. Every MoE layer also adds all-to-all dispatch and combine traffic that can consume those gains. The Expert Parallel Load Balancer can redistribute hot and cold experts, but communication still has a cost. Treat the memory pool as coordinated placement and access; remote HBM retains its own latency and bandwidth constraints.

Dynamo makes reuse a scheduling decision

Dynamo coordinates reuse of KV state that has already been computed. The source describes routing that considers both cached-prefix overlap and worker load, separate GPU pools for prefill and decode, and a KV Cache Runner that tracks blocks across GPU, pinned host memory, and storage tiers. NIXL supports data transfer, while the inference engine retains control of GPU memory and local host offloading. Expanding cache capacity therefore also requires serving coordination. Asynchronous movement alone does not establish that a returning request will avoid waiting. NVIDIA Technical Blog

Our scheduling criterion is a total-time comparison: lookup, queueing, transfer, and any required conversion on the request's critical path must cost less than repeating prefill for the same prefix segment. A higher hit rate can still coincide with longer queues at a popular worker. Reuse across instances also requires checking model weight versions, input tokens and positional semantics, and cache-format compatibility. Identical document text does not guarantee identical KV state. These caches preserve intermediate computations; treating them as freely interchangeable semantic memory between agents would require additional assumptions that the source does not establish.

Vera and CMX extend the places context can wait

During a tool call, a paused session's KV cache can occupy HBM while generating no tokens. NVIDIA says each Vera CPU offers up to 1.5 TB of LPDDR5X and connects to Rubin through 1.8 TB/s NVLink-C2C, providing a place to stage that context. The link figure is a specification; actual restoration speed also depends on access patterns and scheduling. Faster CPU execution may help tools dominated by local computation. When a remote API determines the wait, there is no basis for assuming that the gap will shrink by a corresponding amount.

CMX adds a more distant tier: Ethernet-attached flash shared across a pod, described as G3.5 and built around BlueField-4 STX, Spectrum-X, DOCA Memos, and Dynamo/NIXL integration. This provides capacity for reuse across nodes, with prestaging into GPU or host memory intended to reduce stalls. Our inference is that longer waits and a higher probability of reuse are more likely to justify the movement overhead. Sessions that return quickly, and prefixes unlikely to be reused, may gain little from a trip through these tiers. NVIDIA Technical Blog

Validate the pauses as well as the tokens

Consider a coding agent that reads repository content, generates a tool call, waits for tests, and resumes with the results. Under memory pressure, a lower tier could preserve its reusable prefix while the paused session releases HBM, then restore it when needed. The hypothesis to test is that long prefixes, frequent reuse, and sufficiently long tool gaps amplify the benefit. Short requests, little reuse, or congested links may make migration uneconomic. A cloud agent evaluation should replay real pauses and resumptions; continuous decoding alone does not exercise this pattern.

Hold the model, request trace, context distribution, and latency target fixed. Where the platform supports it, toggle quantization, expert parallelism, cache routing, and tiering separately before evaluating their combination. Record task correctness, time to first token, session-resumption latency, tail queueing latency, cache hits, and bytes moved. For comparisons between generations, use consistent workloads and power-accounting boundaries that include CPUs, networking, and storage. The source's separate storage-throughput and first-token tests use other hardware or software stacks, so they do not independently reproduce the complete Rubin result. Maximum gains from different tests cannot simply be multiplied.