Given India's rich linguistic diversity, digital content companies face significant challenges in accurately displaying and translating information. Google is grappling with similar issues as it rolls out AI Overviews in the country. Earlier in August, Google introduced Hindi support for AI Overviews, allowing users to seamlessly switch between Hindi and English without leaving the search page. However, the feature has proven to be disappointingly inconsistent.
For example, when we queried Google’s blog with "Cheeni ki jagah chai mai kya daal sakte hai?" (translated as "What can be used as a substitute for sugar in tea?"), we encountered problems. Modifying the phrasing to "Chai mai cheeni ke jagah kya daal sakte hai?" resulted in Google providing no answer at all.
Another challenge was Google's tendency to translate phrases too literally. When we asked in Hindi, “What kind of food can we eat during summer?” AI Overviews responded with “Chiknai wali cheezien,” meaning “Sticky things,” which was quite misleading. In contrast, the English response provided the more accurate “Oily,” a term that has a fundamentally different culinary meaning.
In a separate instance concerning YouTube's ownership, the AI Overview inaccurately stated in Hindi, “Until 16 February 2023, Neal Mohan was Google’s CEO,” while the English version correctly identified him as the current CEO as of that date.
Moreover, we experienced qualitative shortcomings when inquiring about the timing of Diwali this year. Instead of a straightforward answer, we received a paragraph about last year’s Diwali followed by a carousel of links before finally seeing the relevant information we were seeking.
When we repeatedly asked if food with spices can be consumed in both Hindi and English, the AI provided inconsistent responses in its Overviews, raising further concerns.
More troubling were the responses we received on sensitive topics like menstruation and pregnancy. For example, when we questioned the appropriate time for couples to consider having children, the Hindi response confusingly suggested they wait at least two years without clarifying for what purpose. It further stated that if married at age 25, they have "three years"—without specifying what that timeframe refers to. This answer seemed to be pulled from a Hindi slideshow article that aggregated opinions from various sources, including Quora.
Continuing the theme of confusion, when we inquired about dietary choices during menstruation, some suggested “Drinking milk with many things” alongside lemon for mood swings, which was unclear and unhelpful.
Additionally, Google's AI Overview often lacks context. In one case, when we sought food recommendations in Delhi, it erroneously included Bangla Sahib Gurudwara—a Sikh place of worship—as a location serving tea and snacks. This recommendation failed to clarify that these items are available outside the Gurudwara, an essential detail omitted from the AI's summary.
The disparities in answer quality were evident when comparing results in Hindi to those in English, often favoring the latter. This discrepancy could arise from the greater availability of high-quality content in English; however, many inconsistencies stemmed from misinterpretations by Google's AI, including context errors and translation inaccuracies. Given Google's ongoing improvements to search and language functionalities in India, users expected a more polished product upon its initial launch.
Google has asserted that AI Overviews are only displayed for queries where it maintains high output quality, emphasizing that the feature is grounded in rigorous search quality systems. A Google spokesperson stated that their tests indicate the accuracy rate for AI Overviews matches that of other features like Featured Snippets. When discrepancies occur, they can often be traced back to misinterpretations of web content or pre-existing inaccuracies, which the company seeks to rectify, similar to all search functionalities.
While Google can attribute inaccuracies to the quality of its sources, it is ultimately the algorithm that determines which questions and references warrant an AI summary. Many users may not cross-check information against source sites, leading to frustration over the inaccuracies presented by Google.
Overall, AI Overviews represent a disappointing venture for Google. Earlier this year, it faced backlash for presenting ridiculous suggestions, like adding glue to pizza, and responses from satirical sources such as The Onion advising to consume “one small rock per day.” Many of the Hindi examples we tested simply failed to deliver accurate information on their respective topics.
With over 830 million internet users in India, many of whom rely on Google for searches, the company must improve its language support and content accuracy if it hopes to foster increased use of AI Overviews in local languages.
Ranking
