AI2 Unveils Largest Open Dataset for Advancing Language Model Training

Unlocking Language Models: AI2’s Open Data Initiative with Dolma

Language models like GPT-4 and Claude showcase remarkable capabilities, yet the exact datasets they are trained on often remain enigmatic. The Allen Institute for AI (AI2) is addressing this issue with the launch of Dolma, a comprehensive, freely accessible text dataset designed for inspection and use by the AI research community.

Dolma serves as foundational data for AI2’s forthcoming open language model, known as OLMo (with Dolma standing for "Data to Feed OLMo’s Appetite"). By making the dataset openly available, AI2 believes it enhances transparency and fosters innovation among AI researchers who aim to build and refine these models.

This marks AI2’s first public release related to OLMo, with a detailed blog post from Luca Soldaini explaining the selection of sources and the methodologies employed to ensure the dataset's suitability for AI applications. They also hint that a more detailed research paper is underway.

While organizations like OpenAI and Meta share some statistics about their training datasets, much of this information is often treated as proprietary. This secrecy has led to concerns about the ethical and legal acquisition of data, raising speculation that some datasets may include unauthorized content, such as pirated books.

To illustrate this point, AI2 created a chart revealing the limited information disclosed by major models, leaving many researchers in the dark about crucial dataset elements. Questions arise regarding the omitted data and how factors like text quality and personal information management were handled.

Although companies may choose to protect their dataset methodologies in a competitive AI landscape, this practice creates barriers for external inquiry and replication of research. In stark contrast, AI2’s Dolma is fully transparent, detailing its sources and processes—such as the decision to include only original English texts—openly for scrutiny.

Dolma is not the first initiative to provide an open dataset, but it stands out as the largest—boasting 3 billion tokens—and claims to be the simplest in terms of usage and licensing. It operates under the “ImpACT license for medium-risk artifacts,” which requires users to:

- Provide their contact details and intended applications.

- Disclose any creations that derive from Dolma.

- Share those derivatives under the same license.

- Refrain from utilizing Dolma in areas such as surveillance or misinformation.

For individuals concerned that their personal data might inadvertently be included in the dataset, a request form for data removal is available for specific cases.

If this initiative resonates with you, explore and access Dolma through Hugging Face for comprehensive AI research.