Release of the 2024 Report: Evaluation of Mainstream AI Models in China's Market

On June 12, the International Data Corporation (IDC) published its report, "Assessment of Mainstream Products in China's Large Model Market, 2024." This report adopted a hands-on testing approach, assembling a product testing team to evaluate foundational large models and related products across various dimensions. External experts assessed the accuracy and reasoning of different products, culminating in final evaluations overseen by a review committee.

In the foundational capabilities assessment, large model products showcased high maturity levels in question understanding, reasoning, and creative expression. Baidu's Wenxin large model stood out in evaluations related to multimodality, security, and text style transfer, highlighting its strong foundational capabilities. In tests measuring logic and reasoning, particularly in mathematics and coding, the Wenxin model demonstrated impressive systematic, logical, and abstract thinking skills, emerging as a leading vendor across all sub-dimensions of coding assessment.

Baidu has further developed the intelligent coding assistant Comate based on the Wenxin large model, achieving a 46% adoption rate, with 27% of new code generated via this tool. The application capability assessment evaluated large model products in consumer scenarios, such as office tools and daily assistants, along with specific industry applications for business-to-business (B2B) contexts. Results indicated that Baidu's Wenxin model excelled in various office tasks, including search, email composition, and chart generation, as well as in consumer scenarios related to daily living and creative interactions.

Additionally, the Wenxin model has fostered a diverse application ecosystem across industries like energy, finance, media, healthcare, telecommunications, manufacturing, transportation, and the internet, effectively addressing real-world challenges. The report also noted Baidu Smart Cloud's launch of the Qianfan Large Model Platform, a comprehensive development and service operation platform tailored for enterprise-level large models.

In late May 2024, Baidu announced the availability of two key models from the Wenxin series, ERNIE Speed and ERNIE Lite, at no cost. IDC anticipates that the second quarter of 2024 will usher in significant updates and enhancements in China's foundational large models and products. The IDC China Large Model Product Testing Team recognized a growing industry focus on the practical applications of large models and generative AI, encouraging technology providers to improve generation quality, speed, and cost-effectiveness to accelerate the adoption and growth of large model technologies.