OpenAI has officially introduced Sora, a revolutionary video generation model that is capturing widespread attention across social media. Users are marveling at its capabilities, with Nate Chan exclaiming, “This is insane quality,” while MIT podcaster Lex Friedman described it as “truly remarkable.” Popular YouTuber MrBeast humorously urged OpenAI CEO Sam Altman, “plz don’t make me homeless.” This launch comes at a time when competition in artificial intelligence is accelerating rapidly. On the same day, Google showcased an upgraded version of its large multimodal model, Gemini 1.5, which can process an astonishing one million tokens, allowing for inputs of up to 700,000 words or the equivalent of one hour of video.
Last month, Google also revealed Lumiere, a video generation model celebrated for its remarkable realism. In a recent blog post, OpenAI detailed Sora’s skills, explaining that it can transform text or still images into videos lasting up to one minute while ensuring high visual quality and fidelity to user prompts. The model is capable of depicting various perspectives of a scene and showcasing different emotions of characters.
Notably, OpenAI asserts that Sora comprehends not only user requests but also how these elements interrelate in the real world. For example, consider a scenario where Sora interprets the prompt: "A cat waking up its sleeping owner demanding breakfast. The owner tries to ignore the cat but ultimately reveals a hidden stash of treats under the pillow." Below is a visual example of what Sora generated based on this description.
While Sora is impressive, OpenAI acknowledges certain limitations. The model struggles with understanding cause and effect; for instance, if a person bites into a cookie, the cookie unfortunately remains intact in Sora's output. Additionally, the model can confuse right and left directions. As part of their safety measures, OpenAI engaged hacker groups to identify potential vulnerabilities in Sora prior to its release.
### The Technology Behind Sora
OpenAI describes Sora as a diffusion model, a sophisticated framework that incorporates random noise into datasets and subsequently learns how to reverse this process to create high-quality data samples. It utilizes transformer architecture to further enhance its capabilities.
Sora is designed to generate entire videos in one go or to extend existing videos, maintaining continuity even when subjects momentarily go out of view. The model is trained on smaller data segments referred to as ‘patches,’ akin to tokens in OpenAI’s earlier GPT language models. This innovative approach allows for training diffusion transformers across a broader spectrum of visual data, including varying durations, resolutions, and aspect ratios.
Moreover, Sora builds on the foundations laid by OpenAI's text-to-image model, DALL-E 3, and GPT models. It adopts a recaptioning technique from DALL-E 3, enabling it to produce highly descriptive captions that enhance visual training, thus allowing Sora to respond to prompts with greater accuracy. Additionally, Sora can accept existing video inputs, enabling users to extend footage or complete missing frames.
OpenAI has not confirmed whether Sora will be integrated into ChatGPT, as was done with DALL-E 3, to enhance the chatbot’s multimodal capabilities. Conversely, Google’s Gemini language model is designed to be multimodal from the outset. In a forward-looking statement, OpenAI notes that “Sora serves as a foundation for models that can understand and simulate the real world, a capability considered pivotal for advancing toward artificial general intelligence (AGI).”
Expect further insights in an upcoming technical paper that will delve deeper into Sora's functionalities and implications.
Ranking
