In a significant development for AI inference, NVIDIA has unveiled its TensorRT-LLM multiblock attention feature, which substantially enhances throughput on the NVIDIA HGX H200 platform. According to NVIDIA, this innovation boosts throughput by more than 3x for long sequence lengths, addressing the increasing demands of modern generative AI models.

Advancements in Generative AI

The rapid evolution of generative AI models, exemplified by the Llama 2 and Llama 3.1 series, has introduced models with significantly larger context windows. The Llama 3.1 models, for instance, support context lengths of up to 128,000 tokens. This expansion enables AI models to perform complex cognitive tasks over extensive datasets, but also presents unique challenges in AI inference environments.

Challenges in AI Inference

AI inference, particularly with long sequence lengths, encounters hurdles such as low-latency demands and the need for small batch sizes. Traditional GPU deployment methods often underutilize the streaming multiprocessors (SMs) of NVIDIA GPUs, especially during the decode phase of inference. This underutilization affects overall system throughput, as only a small fraction of the GPU’s SMs are engaged, leaving many resources idle.

Multiblock Attention Solution

NVIDIA’s TensorRT-LLM multiblock attention addresses these challenges by maximizing the use of GPU resources. It breaks down computational tasks into smaller blocks, distributing them across all available SMs. This not only mitigates memory bandwidth limitations but also enhances throughput by efficiently utilizing GPU resources during the decode phase.

Performance on NVIDIA HGX H200

The implementation of multiblock attention on the NVIDIA HGX H200 has shown remarkable results. It enables the system to generate up to 3.5x more tokens per second for long-sequence queries in low-latency scenarios. Even when model parallelism is employed, resulting in half the GPU resources being used, a 3x performance increase is observed without impacting time-to-first-token.

Implications and Future Outlook

This advancement in AI inference technology allows existing systems to support larger context lengths without the need for additional hardware investments. TensorRT-LLM multiblock attention is activated by default, providing a significant boost in performance for AI models with extensive context requirements. This development underscores NVIDIA’s commitment to advancing AI inference capabilities, enabling more efficient processing of complex AI models.

Image source: Shutterstock