vLLM Support for NVIDIA Vera Rubin NVL72: 7.8x Throughput over GB200 NVL72

vLLM now supports the NVIDIA Vera Rubin NVL72 platform, enabling agentic inference. This support has been developed through collaboration between NVIDIA, Inferact, Red Hat, and the vLLM community. The Vera Rubin platform features hardware enhancements, including 5x the NVFP4 FLOPS compared to GB200 NVL72 and 2.4x higher memory bandwidth in its HBM4 implementation. Performance improvements stem from Rubin's design, which enhances softmax throughput and provides access to superior memory bandwidth via HBM4. The vLLM community has implemented specific optimizations, such as Rubin-tuned kernels for attention, GEMM, and MoE, along with locality-aware memory management using CUDA 13.4 locality domains to optimize MoE performance on the hardware. Early benchmarks show substantial gains, achieving 7.8x throughput per GPU compared to GB200 NVL72 in some scenarios and up to 3.7x higher VLM throughput in MLPerf testing.

The narrative centers on the iterative process of adapting high-performance inference software (vLLM) to bleeding-edge, co-designed hardware architectures (Vera Rubin). The focus shifts from raw computational power to optimizing data locality and kernel design—specifically through exploiting CUDA's locality domains for Memory-bound workloads like MoE decode. This suggests a recurring pattern in AI infrastructure development: maximizing performance is less about brute-force compute scaling and more about intelligently managing the memory hierarchy. The progression from standard Blackwell kernels running on Rubin to specialized, tuned kernels shows an emergent need for deep integration between the software framework and hardware features to unlock novel potential. A critical question arises regarding the long-term maintenance of this specialized optimization layer; if future hardware innovations bypass these specific CUDA features, how resilient is the achieved performance gain? Furthermore, the development pathway suggests a dependency on a tightly knit ecosystem of contributors (NVIDIA, Inferact, Red Hat, vLLM community) to translate architectural leaps into usable, performant tooling. The focus on locality domains implies that future advancements in performance will likely depend on standardized, portable methods for cross-architecture memory management rather than architecture-specific tuning.