Zuckerberg: Meta Requires 10x More Computing Power for Training Llama 4 Compared to Llama 3

Meta, renowned for developing one of the largest foundational open-source large language models, Llama, has acknowledged the need for significantly more computing power to train future models. During Meta’s second-quarter earnings call, Mark Zuckerberg highlighted that training Llama 4 will require ten times the computing resources used for Llama 3. He emphasized the importance of enhancing Meta’s capacity to train models to keep pace with competitors.

“The computing power necessary for Llama 4 will likely be almost 10 times what we utilized for Llama 3, and this demand will continue to escalate for future iterations,” Zuckerberg stated. “While predicting further trends over multiple generations is challenging, I believe it's wiser to invest in building our capacity ahead of demand rather than waiting too long, considering the lengthy lead times for initiating new inference projects.”

Meta launched Llama 3, which boasts 8 billion parameters, in April. Recently, they introduced an upgraded version known as Llama 3.1 405B, featuring 405 billion parameters, marking it as Meta’s most extensive open-source model to date.

Meta’s CFO, Susan Li, also mentioned that the company is exploring various data center projects and expanding capacity to train upcoming AI models. She indicated that these investments are expected to increase capital expenditures in 2025.

Training large language models comes with significant costs. Meta’s capital expenditures surged nearly 33%, reaching $8.5 billion in Q2 2024, compared to $6.4 billion the previous year. This rise is attributed to substantial investments in servers, data centers, and network infrastructure. As reported by The Information, OpenAI allocates $3 billion for training models, in addition to $4 billion for renting servers at a discounted rate from Microsoft.

“As we expand the training capacity for generative AI to enhance our foundational models, we will continue to develop our infrastructure flexibly, allowing us to allocate training resources effectively over time. This will enable us to direct capacity to generative AI inference or to our core ranking and recommendation systems, based on what we see as most valuable,” Li explained during the call.

Furthermore, Meta highlighted the impressive usage of its consumer-facing Meta AI, noting that India has emerged as the largest market for its chatbot. Nevertheless, Li cautioned that the company does not anticipate significant revenue contributions from its generative AI products in the near future.