"MLPerf 4.0 Training Results: Discover Up to 80% Boost in AI Performance Gains"

Innovation in Machine Learning and AI Training Accelerates

Advancements in machine learning (ML) and artificial intelligence (AI) training are rapidly evolving, particularly with the emergence of more sophisticated generative AI tasks.

Today, MLCommons unveiled the MLPerf 4.0 training benchmark, showcasing record-breaking performance levels. This vendor-neutral standard is widely recognized within the industry, featuring contributions from 17 organizations and over 205 results. This release marks the first update in MLPerf training since version 3.1 in November 2023.

The MLPerf 4.0 benchmarks encompass significant developments, including image generation using Stable Diffusion and Large Language Model (LLM) training for GPT-3. Notable first-time results include a new LoRA benchmark that fine-tunes the Llama 2 70B language model for document summarization with a focus on parameter efficiency.

When comparing results to the last cycle, the gains are remarkable.

“Relative to six months ago, some benchmarks have shown nearly 2x performance improvement, particularly with Stable Diffusion,” said MLCommons founder and executive director David Kanter during a press briefing. “That’s impressive for just half a year.”

Specifically, Stable Diffusion training is 1.8x faster compared to November 2023, while GPT-3 training sees a speed increase of up to 1.2x.

AI Training Performance: Beyond Hardware

While hardware plays a significant role in AI model training, software and network connectivity within clusters are equally critical.

“AI training performance hinges on various levers that improve efficiency,” Kanter observed. “The distribution of tasks and communication between multiple processors or accelerators is vital.”

Vendors are not only capitalizing on superior silicon but are also leveraging advanced algorithms and scaling for enhanced performance over time.

Nvidia's Leadership in Training with Hopper Architecture

Nvidia has notably excelled in the MLPerf 4.0 benchmarks, achieving new performance records in five out of nine tested workloads. Impressively, these benchmarks were primarily set using the same core hardware platforms as in June 2023.

David Salvator, director of AI at Nvidia, emphasized the continued value of the H100 Hopper architecture.

“Throughout Nvidia’s history, we typically achieve 2x to 2.5x performance improvements thanks to software innovations during a product lifecycle,” he stated.

Nvidia has employed multiple strategies to boost performance for MLPerf 4.0, including full-stack optimization, finely-tuned FP8 kernels, and an optimized cuDNN FlashAttention.

Importance of MLPerf Training Benchmarks for Enterprises

MLPerf benchmarks offer organizations standardized metrics on training performance, but their value extends beyond mere numbers.

Salvator highlighted that performance enhancements are achieved with existing hardware, proving that Nvidia can derive sustained benefits from established architectures. As organizations plan new deployments, particularly on-premises, the potential for ongoing improvements after initial investment is crucial.

“In terms of performance's significance, the simple answer is that it drives return on investment for businesses,” he concluded.