A week after unveiling an array of enhancements for its suite of Copilot AI-driven products, Microsoft is rolling out exciting new Copilot features on Windows for all users. These features include a tool designed to understand and respond to queries based on the content displayed on your screen.
Today, refreshed Copilot applications for iOS, Android, Windows, and the web are making their debut. Each app features a Copilot with a more “friendly” and “distinct” tone, as described by Microsoft. Additionally, users can now interact with Copilot via WhatsApp, allowing for direct messages in a manner akin to other bots on Meta’s messaging platform.
Copilot Vision
Copilot Vision offers a unique perspective on your PC activities—specifically, it provides insights into the websites you’re visiting on Microsoft Edge. Available through Copilot Labs, a new opt-in program for exclusive experimental features, Copilot Vision can analyze text and images on web pages and respond to questions such as, “What’s the recipe for the food in this picture?”
To access Vision, simply type “@copilot” in Edge’s address bar. While it’s not groundbreaking technology—Google offers comparable search capabilities on Android and has integrated similar features into Chrome—Microsoft claims Copilot Vision is more powerful and prioritizes user privacy.
According to Microsoft, “Copilot Vision can suggest next steps, answer questions, and assist with tasks, all while you converse with it in natural language.” For example, if you’re furnishing a new apartment, Copilot Vision can help you search for furniture, select color palettes, and explore layout options.
Eager to avoid further scrutiny regarding AI privacy, Microsoft highlights that Copilot Vision is designed to delete data immediately after interactions. The company asserts that processed audio, images, or text are not stored or used in training models, at least in this preview phase.
However, it’s important to note that Copilot Vision has restrictions on the types of websites it can interact with. Currently, it is not available for paywalled or “sensitive” content, confining its functionality to a pre-approved list of popular websites.
What qualifies as “sensitive” content is still unclear. Whether it pertains to adult themes or violent imagery, Microsoft has not provided specific details.
Recent legal challenges have put Microsoft in a tricky position concerning AI and paywalls. The New York Times has claimed that Microsoft enabled users to bypass its paywall by serving articles through the Copilot chatbot on Bing, returning nearly verbatim snippets of paid content when prompted correctly.
Copilot Vision, only available in the U.S. for now, will respect sites’ “machine-readable controls on AI,” which include regulations preventing bots from scraping data for AI training. Microsoft has yet to disclose exactly which controls will be respected, and many major publishers have opted to block AI tools from their sites out of concern for data misuse.
Microsoft assures users that it is dedicated to addressing feedback and concerns. “Before we launch more broadly, we will refine our safety measures and prioritize privacy and responsibility,” the company noted in a recent blog post. “There is no specific processing of the websites you are browsing with Copilot; it simply reads and interprets images and text on the page in real-time.”
Think Deeper
Like Vision, the new Think Deeper feature aims to enhance the versatility of Microsoft's assistant. Think Deeper equips Copilot with the ability to tackle more complex tasks, thanks to advanced “reasoning models” that provide step-by-step solutions after thoughtful consideration.
Microsoft was somewhat vague when asked about which models are being utilized, only mentioning that Think Deeper employs “the latest models from OpenAI, fine-tuned by Microsoft.” By inference, it seems likely that the models in use are customized versions of OpenAI’s o1 model.
“This feature is designed to assist with practical everyday challenges,” Microsoft explained. “Think Deeper can help with tasks ranging from solving intricate math problems to comparing the costs of home improvement projects.”
While Microsoft has emphasized the potential of Think Deeper, if it is based on the o1 model, it may have limitations in various areas. Users are curious to learn about the specific enhancements Microsoft has made to the underlying technology and how transparent Think Deeper will be about its own limitations.
For now, Think Deeper will be accessible to a select group of Copilot Labs users in Australia, Canada, New Zealand, the U.S., and the U.K.
Copilot Voice
Today marks the general availability of Copilot Voice (different from GitHub’s Copilot Voice). Initially launching in English in New Zealand, Canada, Australia, the U.K., and the U.S., this feature introduces four synthetic voices, enabling users to converse with Copilot and receive spoken responses.
Similar to OpenAI’s Advanced Voice Mode for ChatGPT, Copilot Voice is adept at recognizing your tone and adapting its responses accordingly. You can interject at any point while Copilot Voice responds. Microsoft's spokesperson mentioned that this mode utilizes “the latest voice technology with new models specifically tuned for the Copilot app,” but specifics on technology remain undisclosed.
It's also crucial to note that Copilot Voice has time-based limits on usage. While Pro subscribers receive more minutes, the precise allocation depends on demand.
Personalization
Microsoft is further enhancing Copilot to be more attuned to individual preferences with a new personalization setting. When enabled, Copilot will leverage your previous interactions and history, as well as your engagement with other Microsoft applications, to suggest tailored ways to utilize the assistant.
“This feature helps you quickly get started,” Microsoft mentioned in a blog post, “offering a helpful guide to Copilot’s features and potential conversation starters.”
The personalization feature can be turned off in the Copilot settings menu on Windows but is not expected to be available for users in the U.K. or EU in the near future. However, other users should start experiencing this feature today.
Microsoft's relationship with EU regulations has been complex, especially regarding the rollout of AI products. In May, the EU warned Microsoft of potential fines for failing to comply with requests for information about its generative AI tools. Many tech giants, including Apple and Meta, have adopted a cautious stance in deploying AI tools in the EU, wary of potential violations of privacy laws.
“As we assess options for users in the European Economic Area and a limited selection of other countries, we plan to roll out Copilot personalization carefully,” a Microsoft spokesperson told us. “Certain features will not be available in the EEA until a later date.”
Ranking
