Around the beginning of this century, artificial intelligence (AI) laboratories discovered that increasing the scale of algorithms and models, along with providing them with more data, significantly enhances their performance. The latest AI models now boast hundreds of billions to trillions of internal network connections, learning to write code like humans by tapping into vast resources from the internet. However, training these larger algorithms demands increasingly powerful computing capabilities. Data from the nonprofit AI research organization EpochAI shows that the computational power dedicated to AI training has been doubling every year. If this trend continues until 2030, future AI models could possess computational power 10,000 times greater than today's leading algorithms.
EpochAI states, "If this trend persists, we could witness enormous advancements in AI by the end of this decade, comparable to the leap from GPT-2's simplistic text generation in 2019 to GPT-4's complex problem-solving abilities in 2023." Nevertheless, modern AI consumes significant amounts of electricity, advanced chips, and trillions of online instances. The industry has already faced chip shortages, and research indicates it may run low on high-quality training data. Given the continued investments in AI, can such growth realistically be sustained? EpochAI explores four key constraints to AI scalability: power, chips, data, and latency.
1. Power: The Need for Massive Electricity
According to EpochAI, training an advanced AI model by 2030 will require electricity equivalent to the annual usage of 23,000 American households—about 200 times more than currently needed. This would represent approximately 30% of the total energy consumed by all data centers today. Few power plants can supply this amount of energy, and many are under long-term contracts. Many businesses will be seeking regions where they can source power from multiple nearby power plants through the local grid. While this presents challenges, especially with planned utility expansions, it remains feasible.
To alleviate power constraints, companies can distribute training across multiple data centers, transferring training data in batches to reduce the energy load on any single center. This approach necessitates fast, high-bandwidth connections—technically achievable, as demonstrated by Google’s Gemini supercomputer trials. EpochAI outlines potential power requirements ranging from 1 megawatt (local power) to 45 megawatts (distributed power). The more power companies harness, the larger the models they can train. In scenarios with limited electricity, models could theoretically be trained with approximately 10,000 times the computational capacity of GPT-4.
2. Chips: Meeting Computational Demands
All this power is utilized to operate AI chips, with some providing complete AI models to customers, and others training upcoming models. EpochAI focuses particularly on the latter. AI labs use graphics processing units (GPUs) for training, with NVIDIA being a leader in this space. The production of these chips is managed by companies like TSMC, which pairs them with high-bandwidth memory. Predictions must account for these three interconnected processes. EpochAI suggests that while GPU production may have some excess capacity, developments in memory and packaging might stymie growth. Current forecasts indicate that by 2030, there could be between 20 million and 400 million AI chips available for training. A substantial number of these will serve existing models, leaving AI labs with limited access.
3. Data: The Online Hunger for Information
AI's insatiable thirst for quality data, coupled with a looming scarcity, poses a significant challenge. Some predict that high-quality public data sources could be exhausted by 2026. However, EpochAI believes that data shortages won't impede model development at least until 2030. Their analysis indicates that, at the current growth trajectory, AI labs could deplete their high-quality text data supply within five years, and evolving copyright litigation could further complicate availability. Yet, even if court rulings favor copyright holders, complex enforcement and licensing agreements from companies like VoxMedia and Time magazine suggest that the impact on data supply will be limited.
Importantly, current models are not solely reliant on text data. For instance, Google’s Gemini incorporates images, audio, and video data in its training. Non-text data can supplement text supplies, such as through subtitles and scripts. Moreover, non-text data may expand models' capabilities, like identifying contents in a fridge and suggesting meals. Interestingly, there’s speculation about "transfer learning," where models trained on diverse data types outperform those trained on a single type. EpochAI also acknowledges that synthetic data has the potential to further enhance the data pool, though the exact benefits remain unclear. For example, DeepMind has long employed synthetic data in its reinforcement learning algorithms, and Meta uses some in its latest AI models. However, the quantity of synthetic data that can be utilized without degrading model quality may be constrained, and its generation requires costly computational resources. Overall, EpochAI estimates that a sufficient amount of combinatory data—incorporating text, non-text, and synthetic data—exists to train AI models with up to 80,000 times the computational power of GPT-4.
4. Latency: The Trade-off of Scale
The final limiting factor is tied to the scale of upcoming algorithms. As models grow in size, the time required for data to traverse their artificial neural networks also increases, which can render the training of new algorithms impractically slow. EpochAI assessed the potential size of future models, the batch size of training data for parallel processing, and the time needed for data processing within and between AI data center servers. It estimates that, under current setups, training AI models could face limitations on scaling but should not persist for long. EpochAI suggests that utilizing existing models could allow for training at 1 million times the computing power of GPT-4.
Scalability: A 10,000-Fold Potential
Notably, in every identified constraint, the potential scale of AI models increases—indicating that chip limits exceed power limits, while data limits surpass chip limits, and so on. However, when all constraints are considered concurrently, models can achieve only the scale determined by the first encountered bottleneck; in this case, that bottleneck is power. Despite this, significant technological scalability remains achievable. EpochAI states, "Considering all these AI bottlenecks, training operations could feasibly reach 2e29 FLOP by the end of this decade." This would represent about a 10,000-fold increase over current models, suggesting that the steady trend of expansion could continue uninterrupted through 2030.
While these factors indicate that sustained scalability is technically possible, they hinge on a fundamental assumption: that investments in AI will expand accordingly to finance this growth, and that this expansion will yield impressive and practical advancements. Currently, there are various signals suggesting that tech companies will continue to pour record amounts of cash into AI. Amidst this drive, expenditures for new technologies are reaching heights not seen in years. Alphabet's CEO, Sundar Pichai, acknowledged during last quarter's earnings call that the risks of under-investing far outweigh the risks of over-investing in this climate.
However, further increases in spending are essential. Anthropic's CEO, Dario Amodei, estimates that today's model training costs could reach up to $1 billion, with next year's potentially nearing $10 billion, and approaching $100 billion per model in subsequent years. While this presents staggering figures, businesses may be willing to bear these costs. Reports suggest that Microsoft has invested substantial funds into its Stargate AI supercomputer as part of its collaboration with OpenAI, slated for release in 2028.
The willingness to invest billions or even hundreds of billions does not guarantee success, particularly when those sums surpass the GDP of many countries and consume a significant portion of tech giants' annual revenues. As the initial excitement surrounding AI begins to fade, the sustainability of its growth may pivot to a "What have you done for me lately?" mindset. Investors are increasingly focused on profit margins, and the current investments pale when compared to returns. To justify increased spending, companies must demonstrate that their scaling efforts can produce increasingly powerful AI models. This raises the stakes for upcoming models that must deliver beyond incremental improvements.
If returns diminish, or if a significant number of consumers refuse to pay for AI products, the situation could shift. Critics argue that large language models and multi-modal systems might actually represent an expensive dead-end. Yet, breakthroughs can always occur, suggesting that tasks could be achieved with fewer resources, akin to how our brains efficiently learn on just the energy of a light bulb, far less than the vast data requirements of the internet. Notwithstanding these challenges, EpochAI claims that if current approaches "can automate a significant portion of economic tasks,” the potential economic returns could reach trillions of dollars, justifying the expenses involved. Many industry insiders are willing to take that bet, but the final outcomes remain uncertain.
Ranking
