A groundbreaking open-source language model has now claimed the title of the best in the world, as per the latest rankings from Hugging Face, a leading platform for natural language processing (NLP) research and applications.
The model, named “Smaug-72B,” was publicly released today by Abacus AI, a startup focused on solving complex challenges in artificial intelligence and machine learning. Smaug-72B is a fine-tuned version of “Qwen-72B,” another prominent language model introduced just months ago by a research team at Alibaba Group.
Significantly, Smaug-72B outperforms OpenAI's GPT-3.5 and Mistral Medium—two of the most advanced proprietary language models—across several key benchmarks. Notably, it also exceeds Qwen-72B by a substantial margin in many evaluations.
According to the Hugging Face Open LLM leaderboard, which assesses the performance of open-source language models on various natural language tasks, Smaug-72B is now the first and only open-source model with an average score exceeding 80 across all major evaluations. While it has not yet reached the 90-100 point average indicative of human-level performance, its release signals a potential shift in the open-source AI landscape, suggesting that it may soon rival the capabilities of major tech companies long regarded as inaccessible.
The Open-Source Advantage
“Smaug-72B from Abacus AI is now leading the LLM leaderboard as the first model to achieve an average score of 80,” said Abacus AI CEO Bindu Reddy in a post on X.com. “Our next goal is to publish these techniques as a research paper and apply them to top Mistral models, including Miqu, a 70B fine-tuned version of LLama-2. The techniques we employed specifically target reasoning and math skills, which accounts for the impressive GSM8K scores! We will provide more insights in our upcoming paper.”
Since its release, Smaug-72B stands out not only for its overall performance but also for its exceptional capabilities in reasoning and math tasks—enhanced by the specific fine-tuning techniques applied by Abacus AI. These techniques address common weaknesses in large language models, leading to improved performance.
Other noteworthy open-source developments include Qwen 1.5, a suite of small yet powerful language models ranging from 0.5B to 72B parameters, released by Qwen. Qwen 1.5 surpasses popular proprietary models like Mistral Medium and GPT-3.5, featuring a 32k context length and compatibility with diverse tools for rapid local inference. Additionally, Qwen introduced Qwen-VL-Max, a new large vision language model that competes with Google's Gemini Ultra and OpenAI's GPT-4V.
Implications for the Future of AI
The rise of Smaug-72B and Qwen 1.5 has generated excitement and discussion within both the AI community and broader tech circles. Many experts have praised the contributions of Abacus AI and Qwen to open-source AI, highlighting the rapid advancements made over the past year.
“It’s incredible to think that less than a year ago, we were thrilled about models like Dolly,” remarked Sahar Mor, an AI influencer and analyst, on LinkedIn, reflecting on the swift progress in open-source models.
Both Smaug-72B and Qwen 1.5 are available for public access on Hugging Face, allowing users to download, utilize, and modify them at will. Abacus AI and Qwen also intend to submit their models to the llmsys human evaluation leaderboard—a new metric designed to measure language model performance in human-like tasks. They hinted at future projects aimed at producing more open-source models and exploring diverse applications.
Smaug-72B and Qwen 1.5 exemplify the rapid evolution of open-source AI in recent months. They signify a transformative wave of innovation and democratization, challenging the dominance of major tech firms and broadening opportunities for developers and researchers. While the future of Smaug-72B's leadership on the Hugging Face leaderboard remains to be seen, it is clear that open-source AI is gaining substantial momentum.
Correction: February 7, 2024 – An earlier version of this article inaccurately characterized GPT-3.5 as an open-source model. It is proprietary technology. We apologize for the error.
Ranking
