Zyphra Unveils Zamba: A Game-Changing SSM-Hybrid Foundation Model to Enable AI Accessibility on More Devices

Zyphra Technologies is launching a groundbreaking foundation model designed to further decentralize artificial intelligence. Zamba, an open-source AI model with 7 billion parameters, utilizes the company's Mamba blocks and a global shared attention layer. This innovative model aims to enhance intelligence across various devices while significantly reducing inference costs.

AI for Every Device

"Our vision is to create your personal AI," stated Krithik Puthalath, CEO of Zyphra Technologies. "Our mission is to foster better connections among people. While technology and social media promised a more connected and fulfilling world, we have fallen short. We aspire to transform the future of AI."

Puthalath emphasized that the centralization of AI by major companies poses a critical problem. "In the quest for artificial general intelligence, firms like OpenAI and Anthropic have developed monolithic models in the cloud—single models meant for everyone. This approach has limitations, leading to a lack of trust in these systems, making AI feel impersonal. While ChatGPT provides valuable responses, it lacks true memory, personalization, and the ability to adapt over time."

The Value of Smaller Language Models

Zyphra’s 7-billion-parameter model may seem limited compared to larger models from OpenAI, Anthropic, or Meta, which have tens of billions. However, Zyphra's strategy focuses on deploying small language models (SML) to optimize AI integration in everyday devices.

Beren Millidge, co-founder and chief scientist at Zyphra, believes that while their initial model, BlackMamba, with 1 billion parameters served as a proof of concept, 7 billion parameters are ideal for meaningful interactions. "This size allows for local operation on nearly all devices," he explained. In contrast, larger models typically require powerful GPU clusters that are inaccessible to most users, reinforcing Zyphra's commitment to decentralization.

"This is about bringing AI closer to the user," Puthalath added. "By developing smaller, efficient models tailored for specific use cases, we enable real-time responses without relying on cloud infrastructure. This approach not only enhances user experience but also reduces operational costs, allowing for more investment in innovation."

Competing with Established Models

Zyphra asserts that Zamba excels against other open-source models like LLaMA 1, LLaMA 2 7B, and OLMo-7B, outperforming them across various standard benchmarks while utilizing less than half the training data. Although initial tests were performed internally, Zyphra plans to release the model's weights for public evaluation.

When asked about the development of Zamba's architecture, Millidge shared that their approach is rooted in practical intuition about existing model challenges and potential solutions. They also drew inspiration from neuroscience, creating a structure that mimics the brain's functionality. Zamba features a single global memory block made up of Mamba blocks, allowing for efficient information sharing similar to the interaction between the cerebral cortex and hippocampus in the human brain.

Zyphra's process included significant experimentation. "Intuition alone isn't enough," Millidge noted. "We must conduct experiments to discover what works and what doesn't, then iterate accordingly."

The open-source Zamba foundation model is now available on Hugging Face, inviting users to explore its capabilities.