This morning, Google unveiled AlphaCode 2, the latest edition of its code-generating model developed by DeepMind, alongside the introduction of the Gemini generative AI framework. AlphaCode 2, which utilizes a variant of the Gemini system (Gemini Pro) specifically fine-tuned on coding contest data, boasts significant enhancements over its predecessor, according to Google.
In a recent evaluation of programming competitions on the Codeforces platform, AlphaCode 2 demonstrated superior performance, ranking above approximately 85% of contestants on average. This is a notable improvement compared to the original AlphaCode, which only outperformed about 50% of participants in a similar evaluation.
According to a technical whitepaper on AlphaCode 2, Google selected 12 recent contests with over 8,000 participants, focusing on division 2 and the more challenging division ‘1+2,’ totaling 77 problems. Remarkably, AlphaCode 2 successfully solved 43% of these problems within 10 attempts—almost double the rate of the original AlphaCode, which solved 25%.
AlphaCode 2 excels in tackling programming challenges that involve "complex" mathematics and theoretical computer science. One of its impressive capabilities, as explained by DeepMind research scientist Rémi Leblond, is its proficiency in dynamic programming. This technique simplifies intricate problems by repeatedly breaking them down into simpler subproblems. Leblond noted that AlphaCode 2 not only effectively implements this strategy but also recognizes when and where to apply it, overcoming a significant limitation of the original AlphaCode.
“[AlphaCode 2] demonstrates a level of understanding, reasoning, and designing of code solutions before even moving to implementation,” Leblond remarked, highlighting its performance on previously unseen problems.
AlphaCode 2 employs a sophisticated approach to problem-solving. It begins by utilizing a range of “policy models” to generate various code samples for each challenge. Samples that don’t comply with the problem specifications are filtered out, while a clustering algorithm organizes “semantically similar code samples” to eliminate redundancy. Ultimately, a scoring model identifies the best candidate from each of the 10 largest code sample clusters, which forms AlphaCode 2’s solution to the problem.
However, like all AI models, AlphaCode 2 has its limitations. The whitepaper indicates that it requires extensive trial and error, has high operational costs at scale, and relies significantly on the filtering of poorly written code samples. Transitioning to a more advanced version of Gemini, such as Gemini Ultra, may help address some of these issues.
As for the future of AlphaCode 2, which was never officially launched, Eli Collins, VP of Product at DeepMind, hinted at its potential.
“What excites me most about the latest results is that when programmers collaborate with [AlphaCode 2 powered by] Gemini by establishing specific properties for the code, the model's performance improves,” Collins said. “In the future, we envision programmers utilizing powerful AI models as collaborative tools that enhance the entire software development process, from problem-solving to implementation assistance.”
Ranking
