Meta Galactica Author Speaks Out on Model's Challenging Launch Experience

A year ago, just before OpenAI launched ChatGPT, Meta introduced its own large language model—Galactica. This model sparked significant controversy and was pulled just days after its debut. Now, one of the original authors has shared insights into the project’s tumultuous journey.

Ross Taylor, co-founder of the Papers with Code platform, which archives machine learning research papers, was instrumental in developing Galactica, a model trained on scientific literature. Unfortunately, the model's demo encountered serious issues, generating citations for non-existent papers.

In a candid post on X (formerly Twitter), Taylor reflected on the team’s challenges, admitting, “We were overstretched and lost situational awareness at launch by releasing [a] demo” without sufficient quality checks. He acknowledged that while they anticipated potential criticisms, the intensity of their workload blurred their focus.

Despite its strong performance compared to models like Google’s PaLM and DeepMind’s Chinchilla, the demo overshadowed Galactica's capabilities. Taylor explained that the intent behind the demo was to gauge scientific queries posed to large language models. However, he noted, “We should have known better,” recognizing that journalists often tested the model beyond its intended scope.

Interestingly, he attributed some of the backlash to user perceptions, claiming that many treated Galactica as a finished product, even though it was only a demo of a base model. “We put our vision on the site, which misled expectations,” he said. He expressed that while the controversy could have been mitigated, fears surrounding Galactica generating “fake science” were overstated.

At the time, prominent figures like Michael Black from the Max Planck Institute voiced concern, suggesting that Galactica could fuel a wave of deep scientific fabrications. Grady Booch, a key developer of the Unified Modeling Language, criticized the model as “little more than statistical nonsense at scale.” In response, Taylor labeled much of the criticism as "really stupid," affirming that the project could have launched more successfully with careful planning.

Looking back, Taylor stood by the research and even the decision to release a demo, seeing it as an act of openness from a major player in the field. “This unprecedented openness was misguided given the vulnerabilities it exposed,” he acknowledged. Meta’s commitment to an open-source approach has since led to the development of LLaMA and its more advanced counterpart, Llama 2, with insights from Galactica informing these later models.

In closing, Taylor expressed his willingness to undertake the same project again, stating, “Better to do something and regret it than not do anything at all.” He emphasized that while the experience remains painful, the lessons learned have been invaluable.