Nvidia and Microsoft Join Forces to Address a Major Challenge with Copilot+

When Microsoft unveiled Copilot+ PCs a few weeks ago, one question emerged: Why can’t I run these AI applications on my GPU? At Computex 2024, Nvidia finally addressed this inquiry.

Nvidia and Microsoft are collaborating on an Application Programming Interface (API) that will enable developers to run AI-accelerated applications on RTX graphics cards. This includes the Small Language Models (SLMs) integral to the Copilot runtime, which power features like Recall and Live Captions.

With this toolkit, developers can execute applications locally on your GPU, rather than relying solely on the Neural Processing Unit (NPU). This advancement paves the way for more robust AI applications, as GPUs typically offer superior AI processing power compared to NPUs, and broadens the accessibility for PCs beyond the current Copilot+ requirements.

This is a strategic development. Copilot+ PCs currently depend on an NPU that can perform at least 40 Tera Operations Per Second (TOPS), but presently, only the Snapdragon X Elite meets that specification. In contrast, GPUs demonstrate significantly higher AI processing capabilities, with entry-level models achieving 100 TOPS and advanced models exceeding that.

The newly introduced API also enhances the Copilot runtime with retrieval-augmented generation (RAG) capabilities. RAG allows AI models to retrieve specific local information, enabling them to deliver more effective solutions. We previously witnessed RAG functionality showcased in Nvidia’s Chat with RTX earlier this year.

Beyond the API, Nvidia unveiled the RTX AI Toolkit at Computex. Scheduled for release in June, this developer suite integrates a variety of tools and SDKs, allowing developers to fine-tune AI models for specialized applications. Nvidia asserts that utilizing the RTX AI Toolkit can result in models that are four times faster and three times smaller compared to open-source alternatives.

A surge of tools is emerging that empower developers to create tailored AI applications for end users. While some innovations have already been integrated into Copilot+ PCs, we can expect a greater variety of AI applications to surface in the coming year. With the hardware capable of supporting these applications, we now simply need the corresponding software.