Google Photos is set to enhance user experience with an innovative AI feature called Ask Photos, powered by Google’s advanced Gemini AI model. Slated for rollout later this summer, this experimental feature enables users to conduct searches within their Google Photos library using natural language queries, harnessing AI to understand the context and metadata of their images.
Previously, users could search for specific individuals, locations, or objects in their photos. The introduction of natural language processing will transform this search process into a more intuitive experience, allowing users to find content with ease. Google unveiled this enhancement during its annual Google I/O 2024 developer conference.
For example, instead of simply searching for “Eiffel Tower,” you can now ask, “Show me the best photo from each National Park I visited.” The AI evaluates various factors—like lighting, focus, and background clarity—to determine what constitutes the "best" photo. By combining this with the geolocation data and timestamps, the AI can efficiently retrieve images taken in U.S. National Parks.
This new functionality builds on the recent Photo Stacks feature, which groups similar photos and highlights the best amongst them. With over 6 billion images uploaded daily on Google Photos, these features aim to enhance the user experience as digital collections continue to expand.
Moreover, the Ask Photos capability lets users pose questions to receive insightful answers beyond just retrieving the best vacation photos. For instance, a parent could inquire about the themes for their child's last four birthday parties and receive a direct response featuring photos and videos related to the mermaid, princess, and unicorn themes utilized.
This advanced query functionality works because Google Photos not only recognizes the user's keywords but also comprehends broader concepts in natural language—such as “themed birthday party.” The AI’s multimodal abilities also allow it to consider any text present in the photos that may relate to the query.
Demonstrated by CEO Sundar Pichai ahead of the conference, another example involved a user asking the AI for updates on their child's swimming progress. In response, the AI compiled memorable highlights from photos and videos documenting swimming activities over time.
Additionally, this feature allows users to extract text-based information from images. For example, you could capture a photo of your license plate or passport number, then simply ask the AI to retrieve that information later when needed.
Should the AI make mistakes—such as misclassifying a photo—it will learn from user corrections, gradually personalizing its responses for each individual user over time.
When you’re ready to share photos, the AI can assist in drafting captions that summarize the content. Currently, these captions are basic summaries without style options. However, leveraging Gemini's capabilities, a thoughtfully crafted prompt may yield a desired style.
Google has implemented safeguards to ensure that the AI doesn’t respond to inappropriate queries, and they have intentionally excluded potentially offensive content from the model’s training data. Although launching as an experimental feature, Ask Photos may require further adjustments based on user interactions.
Initially, the Ask Photos feature will be available in the U.S. in English, with plans for expansion to additional markets. Currently a text-based feature akin to interacting with an AI chatbot, future updates may enable deeper integration with Gemini on devices like Android.
Google assures users that personal data stored in Google Photos is not used for advertisements, and human oversight of AI conversations is limited to exceptional cases of abuse or harm. Additionally, personal data is not utilized to train any generative AI products, including Gemini.
Stay updated! Sign up for our AI newsletter to receive the latest news directly in your inbox starting June 5.
Ranking
