Elon Musk's xAI Unveils Grok-1.5V: The First Multimodal AI Model

Elon Musk’s xAI has launched its first multimodal model, Grok-1.5 Vision (Grok-1.5V), which can understand not only text but also various visual data, including documents, diagrams, charts, screenshots, and photographs. This model will soon be available to early testers and current Grok users.

According to a blog post by the company, “Grok-1.5V competes with leading multimodal models across multiple domains, such as multi-disciplinary reasoning and visual comprehension of science diagrams, documents, screenshots, and images.”

The announcement follows the recent unveiling of the updated chatbot model, Grok-1.5. xAI showcased seven examples demonstrating Grok-1.5V’s capabilities. These include transforming a whiteboard flowchart into Python code, generating a bedtime story from a child's drawing, explaining memes, converting tables into CSV files, and assessing whether wooden decks need replacement due to rot.

xAI claims Grok-1.5V has outperformed competitor models like GPT-4V, Claude 3 Sonnet, Claude 3 Opus, and Gemini Pro 1.5 in various assessments. The company highlights Grok-1.5V's superior performance on the RealWorldQA benchmark, a new metric developed to assess real-world spatial understanding.

RealWorldQA was trained on over 700 images, each paired with specific questions and answers. The dataset features a range of anonymized images, including those captured from vehicles. xAI plans to release RealWorldQA to the public under a Creative Commons license.

As xAI continues to advance, it aims to rival OpenAI and other industry leaders, following the launch of its chatbot in November 2023. The release of Grok-1.5V comes shortly after xAI made Grok AI open source. However, the company has faced controversy, including allegations that the Grok chatbot provided guidance on illegal activities.

Despite these challenges, xAI remains committed to developing “beneficial artificial general intelligence” with the capacity to understand the universe. The company has announced that it will introduce significant updates to Grok AI’s multimodal understanding and generation capabilities in the months ahead.