Exclusive: Tenyx's AI Breakthrough: Open-Source Llama 3 Model Surpasses GPT-4 Performance

In an exclusive interview, Itamar Arel, founder and CEO of AI startup Tenyx, unveiled a remarkable advancement in natural language processing. Tenyx has fine-tuned Meta’s open-source Llama-3 language model, now called Tenyx-70B, to exceed OpenAI’s GPT-4 in specific domains—marking a first for an open-source model to surpass the proprietary standard.

“We developed a fine-tuning technology that allows us to enhance a foundational model beyond its original training,” Arel explained. “We’re excited about using this approach to enable continual or incremental learning by leveraging redundancies in large models.”

Tenyx's Llama-3 model outperforms GPT-4 in math and coding while exceeding the base Llama-3 model in all capabilities. This achievement, according to Arel, highlights a new era for open-source AI.

Addressing ‘Catastrophic Forgetting’

Tenyx confronts the problem of "catastrophic forgetting," where a model may lose previously acquired knowledge when introduced to new data. By selectively updating a small fraction of the model’s parameters, Tenyx effectively trains on new information without sacrificing existing skills.

“If you change just 5% of the model parameters while keeping the rest intact, you can do this more aggressively without distorting other functions,” Arel noted. This method allows Tenyx to fine-tune the 70-billion-parameter Llama-3 model in just 15 hours using 100 GPUs.

Commitment to Open-Source AI

Tenyx champions open-source AI by releasing their fine-tuned model, Tenyx-70B, under the same license as the original Llama-3. “We believe in open-source models,” Arel stated. “Sharing advances with the community fosters innovation and benefits everyone.”

The applications of Tenyx’s post-training optimization technology are extensive, from developing specialized chatbots to facilitating frequent updates for deployed models, ensuring they stay current with emerging information.

Reshaping the AI Landscape

Tenyx’s breakthrough has significant implications, providing businesses and researchers access to advanced language models without the prohibitive costs of proprietary solutions. This progress could also ignite further innovation in the open-source community as others build upon Tenyx’s success.

“What does this mean for the industry and companies like OpenAI?” Arel pondered. As competition in the AI sector escalates, Tenyx’s fine-tuning of open-source models could redefine industry dynamics and how businesses approach natural language processing.

While the Tenyx-optimized Llama-3 retains some limitations found in the base model—including occasional illogical responses—its enhancements are noteworthy. Arel reported that the model performs with nearly 96% accuracy in math and reasoning, compared to the base model's 85%.

As Tenyx ushers in a new wave of open-source AI innovation, the long-term impact of their breakthrough on the AI ecosystem remains to be seen. However, it is evident that Tenyx has shown that open-source models can compete with and even surpass proprietary counterparts, paving the way for a more accessible and collaborative future in artificial intelligence.