Exploring the Four Key Dimensions of Multi-Modal Performance: Why GPT-4o is Considered the Most Powerful Model

In March of last year, OpenAI launched GPT-4, earning widespread acclaim. Major tech companies like Google and Meta, along with emerging players such as Mistral AI and Anthropic, have since competed to develop their large language models. Now, with the introduction of GPT-4o, a new chapter begins. On May 13, OpenAI unveiled GPT-4o, which CEO Sam Altman hailed as the “best model OpenAI has ever created.” This model supports text, image, speech, and video inputs and outputs, and OpenAI is making it accessible for free to regular users, while offering a 50% discount on API usage for developers.

As of now, GPT-4o primarily focuses on text and images. A recent examination by a journalist from the Daily Economic News highlighted its notable advancements in image recognition. The results reveal significant improvements in response speed, and the model demonstrates impressive accuracy in identifying and interpreting images. However, its performance in summarizing lengthy texts does not significantly exceed that of previous models.

What distinguishes GPT-4o? On May 15, John Schulman, a co-founder of OpenAI, emphasized the importance of post-training in enhancing the model’s performance during a podcast interview.

Image Recognition Capabilities of GPT-4o

The assessment of GPT-4o’s image recognition capabilities covered four key areas: general images, specialized field images, data visuals, and handwriting.

1. General Image Recognition

- Simple Images: GPT-4o accurately described the movements of a Boston Dynamics robot navigating obstacles.

- Complex Comics: The model successfully summarized a multi-panel comic, interpreting the humor and artistic techniques such as anthropomorphism and exaggeration.

2. Specialized Field Images

- Medical Imaging: GPT-4o explained the mechanism of mRNA vaccines using a diagram, despite the absence of explicit labels.

- Real Estate Analysis: The model evaluated a floor plan of a 134 square meter apartment, identifying strengths and weaknesses, though some accuracy adjustments were necessary.

3. Data Visualization Analysis

- In reviewing a mixed data chart, GPT-4o extracted and represented the information graphically with complete accuracy.

4. Handwriting Interpretation and Logic Reasoning

- The journalist tested the model’s handwriting recognition with a logic puzzle. GPT-4o accurately identified the handwritten text and followed the instructions logically to deliver the correct answer.

How GPT-4o Was Developed

The testing highlighted impressive response times and multimodal capabilities, with Altman reiterating that GPT-4o is OpenAI’s finest achievement yet. So, how was this advanced functionality realized? John Schulman explained that post-training enhances model capabilities by further training on large, unlabelled datasets to deepen understanding of language and knowledge. Since its initial release, GPT-4's Elo score has risen significantly, largely due to the effects of post-training.

Furthermore, Jim Fan, a senior research scientist at NVIDIA, noted that advancements in tokenization and architectural design were critical in developing GPT-4o. He suggested that this model may represent an early iteration of GPT-5, which is rumored to be in development.

In conclusion, OpenAI’s strategic introduction of GPT-4o aims to secure a competitive edge in the rapidly evolving AI landscape, particularly against rivals like Google.