Addressing 'Model Collapse': Scientists Urge Caution Against Self-Feeding AI Systems

When you encounter the mythical Ouroboros, it's reasonable to ponder, "This surely can't endure." As a powerful symbol, representing the cycle of self-consumption, it aptly reflects certain challenges in AI development. A recent study highlights a pressing concern: AI may be on the brink of "model collapse" after repeatedly training on its own generated data.

In a research paper published in Nature, a team of British and Canadian scientists, led by Ilia Shumailov at Oxford, reveals that current machine learning models are particularly susceptible to what they term “model collapse.” The introduction of their paper states:

"We discover that indiscriminately learning from data produced by other models causes 'model collapse' — a degenerative process whereby, over time, models forget the true underlying data distribution..."

So, how does this issue arise, and what causes it? The explanation is quite straightforward.

At their core, AI models are designed to recognize patterns. They learn from their training data and then match new prompts to these identified patterns, predicting the most probable subsequent elements. Whether you inquire, “Can you provide a snickerdoodle recipe?” or “List U.S. presidents by age at inauguration,” the model essentially delivers the most likely continuation of your query. This process is somewhat analogous for image generators, albeit with differences.

However, these models tend to favor the most prevalent outcomes. For example, you won't receive an obscure snickerdoodle recipe; you'll get the most popular one. Likewise, if you request an image of a dog, the output is unlikely to depict a rare breed but rather a common one, such as a golden retriever or Labrador.

Now, consider the explosion of AI-generated content online, which newer AI models are likely to ingest and use for training. This situation implies that they will predominantly encounter familiar images, like golden retrievers!

Once these models begin training on this oversaturation of golden retrievers (or generic blog content, artificial faces, or computer-generated music), this becomes their new standard. They may start to believe that 90% of dogs are golden retrievers, leading to an increase in that output with each generation until they lose track of what dogs actually are.

A striking illustration from Nature’s accompanying commentary visually depicts this cycle:

A similar pattern emerges with language models, which prioritize the most common data within their training sets — a practice that is usually beneficial. However, this becomes problematic in the current landscape of an overwhelming amount of AI-generated content available online.

In essence, if models continue consuming each other’s data, potentially without awareness, they risk becoming increasingly bizarre and less intelligent until a state of collapse occurs. Researchers have provided various examples and strategies to counteract this issue, with some even asserting that model collapse is “inevitable,” at least in theory.

Although it may not unfold exactly as their experiments suggest, this possibility should alarm anyone involved in AI development. The diversity and richness of training data are increasingly recognized as the most critical factors influencing model quality. If the data supply dwindles but generating more poses risks of model collapse, does this fundamentally limit the potential of today's AI? If such a decline begins, how will we recognize it? Can we take proactive steps to mitigate this issue?

The good news is that the answer to the last question is likely yes, though this should not diminish our concerns.

Establishing qualitative and quantitative benchmarks for data variety and sourcing would be beneficial, but we have yet to standardize such practices. Implementing watermarks for AI-generated data could help other AIs avoid replicating it, yet finding an effective way to mark images remains a challenge (though a solution has been proposed).

Moreover, companies might be reluctant to share such information, preferring to hoard high-value original and human-generated data, thereby maintaining what Shumailov et al. describe as their “first mover advantage.”

We must take the threat of model collapse seriously to sustain the advantages of training on large-scale, web-sourced data. As LLM-generated content proliferates online, genuine human interaction data will become increasingly valuable.

As the landscape evolves, it may become progressively difficult to train new iterations of LLMs without access to data collected from before the mass adoption of such technology or direct access to substantial human-generated datasets.

This concern adds to the growing list of potential challenges facing AI models and serves as an argument against current methodologies aimed at generating the next-level superintelligence.