Qualcomm has a long history of running artificial intelligence (AI) and machine learning systems on-device, particularly in their camera chipsets. At the Snapdragon Summit 2023, the company unveiled its latest advancements in on-device AI for mobile devices and Windows 11 PCs, featuring the new Snapdragon 8 Gen 3 and X Elite chips. These chipsets are designed with generative AI capabilities from the ground up and support a variety of large language models (LLMs), language vision models (LVMs), and transformer-based automatic speech recognition (ASR) models, including up to 10 billion parameters for Snapdragon 8 Gen 3 and 13 billion for X Elite, all executed completely offline.
With these powerful chipsets, users can run models like Baidu’s ERNIE 3.5, OpenAI’s Whisper, Meta's Llama 2, or Google’s Gecko directly on their devices without needing an internet connection. Qualcomm’s chips excel in processing voice, text, and image inputs.
“It’s crucial to provide comprehensive model support, which is why heterogeneous compute is essential,” said Durga Malladi, Qualcomm's SVP & General Manager of Technology Planning & Edge Solutions. “Our state-of-the-art CPU, GPU, and NPU are utilized simultaneously, allowing multiple models to operate concurrently.”
The Qualcomm AI Engine integrates the Oryon CPU, Adreno GPU, and Hexagon NPU, collectively capable of handling up to 45 TOPS (trillions of operations per second) and processing tokens at impressive speeds—30 tokens per second on laptops and 20 tokens per second on mobile devices. The chipsets utilize Samsung’s 4.8GHz LP-DDR5x DRAM for efficient memory management.
“Generative AI can tackle and resolve complex tasks efficiently,” Malladi noted, pointing to potential use cases such as document summarization, email drafting for consumers, and prompt-based code or music generation for businesses.
Moreover, Qualcomm is enhancing its focus on photography with its Cognitive ISP technology, enabling devices with these chipsets to edit photos in real-time across multiple layers. Users can capture clearer images in low light, erase unwanted objects, or expand image backgrounds, while also authenticating their shots using Truepic photo capture to indicate they are real and not AI-generated.
Having AI predominantly on-device instead of in the cloud offers numerous advantages. Similar to enterprise AI systems that fine-tune general models (like GPT-4) with internal data, local AIs will “gradually get personalized” over time, improving their accuracy and relevance, according to Malladi. The elimination of cloud processing delays allows the X Elite and Snapdragon 8 Gen 3 to run programs like Stable Diffusion and generate images in under 0.6 seconds.
As consumers increasingly interact with devices through spoken commands, these models facilitate a more natural user experience. "Voice becomes a much more intuitive interface for these devices," Malladi stated, emphasizing the transformative potential of this technology.
Qualcomm sees the introduction of on-device AI in mobile devices and PCs as just the beginning, with future iterations of chips aiming to support models with 20 billion-plus parameters. “These models are highly sophisticated, and the use cases are impressive,” Malladi remarked.
Looking ahead, Qualcomm envisions applications in Advanced Driver Assistance Systems (ADAS), where multi-modal data is processed from various sensors, including cameras, radar, and lidar. As these on-device models advance, they could reach 100 billion parameters or more, opening up new possibilities for innovation in AI technology.
Ranking
