Meta has introduced Llama 3, a groundbreaking open-source language model that sets new standards in reasoning, code generation, and instruction following. Dubbed as "the most capable openly available" large language model to date, Llama 3 outperforms similar models from competitors like Google and Anthropic, solidifying its position as a leader in the AI landscape. The Llama series serves as the backbone for various applications and has inspired subsequent models developed by others, including Vicuna and Alpaca.
According to Meta, this latest generation of Llama showcases state-of-the-art capabilities across a diverse range of industry benchmarks. It excels in understanding nuanced language and executes complex tasks such as translation and dialogue generation with remarkable efficiency. Enhancements in scalability and performance empower Llama 3 to tackle multi-step challenges more effectively, with the company boasting that the new model significantly enhances reasoning and instruction-following abilities.
A notable improvement in Llama 3 is its reduced prompt refusal rate, achieved through a refined post-training process that also enhances the diversity of its responses. Llama 3 features two sizes: an eight-billion-parameter model, slightly larger than its smallest predecessor, and a more robust 70-billion-parameter version. Both models support an 8k context length, allowing for the processing of approximately 6,000 words of input.
Businesses can start utilizing Llama 3 immediately, as it is available for download from Meta’s website and can be accessed through popular cloud platforms such as AWS via Amazon SageMaker JumpStart. Additionally, it will be integrated into services including Databricks, Google Cloud, Hugging Face, IBM watsonX, Nvidia NIM, and Microsoft Azure, ensuring a broad range of deployment options. The model is designed to be compatible with hardware from various providers, including AMD, AWS, Dell, Intel, Nvidia, and Qualcomm.
This launch marks just the beginning of Llama 3's capabilities, as Meta is also working on larger versions, including an anticipated “Long” model with enhanced memory and context lengths. Among the upcoming models is an ambitious 400 billion-parameter version, potentially the largest open-source model ever released. Although still in training, initial performance details suggest that this colossal model is already achieving commendable scores on standardized industry benchmarks.
Jim Fan, a senior AI research scientist at Nvidia, commented on the significance of the future 400 billion parameter model, stating it could offer the community open-weight access to a GPT-4-class model. He noted that this model could alter the research landscape and benefit grassroots startups aiming to push the boundaries of AI technology.
Llama 3 is built on a strong architectural foundation, demonstrating exceptional performance across industry benchmarks such as MMLU and HumanEval. The newly released models are pretrained and fine-tuned for specific tasks, leading to superior reasoning and coding capabilities compared to earlier iterations. Impressively, the smaller Llama 3 model outscored competitive models like Google’s Gemma-7B and Mistal’s Mistral-7B Instruct, while the larger version surpassed scores from Google Gemini Pro 1.5 and Anthropic’s Claude 3 Sonnet.
Underneath its impressive performance, Llama 3 benefits from an optimized underlying architecture, implementing improved post-training methods that reduce false refusal rates and enhance response diversity. It uses a decoder-only transformer architecture, which enables more efficient text generation compared to traditional models. Additionally, the introduction of grouped query attention (GQA) boosts the model’s overall efficiency.
In support of responsible AI deployment, Meta has introduced tools like Llama Guard 2 for content filtering based on the MLCommons taxonomy, CyberSec Eval 2 for assessing code generation risks, and Code Shield for real-time filtering of insecure code outputs.
While Meta has not specified the dataset used for Llama 3, it has highlighted that the training set is substantially larger than that of Llama 2, incorporating over 15 trillion tokens from publicly available sources. Notably, more than 5% of this dataset includes high-quality non-English data spanning over 30 languages, though performance in these languages may not match that of English.
Meta has utilized a series of data-filtering pipelines to ensure the quality of the training data, eliminating not safe for work (NSFW) content and assessing overall data validity. Synthetic data generated from the Llama 2 model has also been integrated to enhance text quality.
The infrastructure behind Llama 3's training is equally impressive, relying on custom-built data center-scale GPU clusters, each comprising 24,576 Nvidia H100 GPUs. Meta's commitment to open-source development underscores its belief that transparency fosters better products, accelerates innovation, and contributes positively to the marketplace. Through a community-focused approach, Llama 3 is now accessible on leading cloud, hosting, and hardware platforms, with further expansion anticipated in the near future.
Ranking
