Training data and datasets

Many of the open text datasets in this catalog start from the same source. Common Crawl, a 501(c)(3) nonprofit, crawls the web and freely provides an archive of crawl data collected since 2008. Hugging Face's FineWeb is built by filtering and deduplicating Common Crawl snapshots. Together AI's RedPajama-V2 contains documents from 84 Common Crawl snapshots, with quality signals for part of the corpus and the IDs of duplicate documents so that you can filter it yourself. Essential AI's Essential-Web, also built from Common Crawl, labels every document with subject, page type, complexity, and quality metadata. Ai2's Dolma mixes web content with academic publications, code, books, and encyclopedic material.14675

A dataset's license covers the compilation, not every work inside it. FineWeb and Dolma are released under the Open Data Commons Attribution License, but FineWeb's card says its use is also subject to Common Crawl's Terms of Use, and Dolma's card says you are also bound by the licenses and terms of the original data sources. RedPajama-V2 points to Common Crawl's terms for its data and uses Apache 2.0 only for its code. Common Crawl's terms in turn say you must evaluate and bear the risks of using crawled content, recommend legal advice before any use, including commercial use, and require you to respect the copyrights and other rights of third parties in that material.4562

Some datasets come with use restrictions instead. Downloading Nemotron-CC-v2 from Hugging Face requires accepting the NVIDIA Data Agreement for Model Training and confirming that the data will be used only for model training. The agreement makes the data available solely for internal training of your own AI models, bars redistributing it, and says that NVIDIA does not grant any rights to copyrighted material the datasets may contain.89

Documentation does not guarantee lasting availability. EleutherAI's Pile was described in a paper and a datasheet, and its card says the license depends on which of its component datasets you use. A 2025 paper describing the Common Pile, a dataset of public domain and openly licensed text, says that the use of unlicensed training data has previously resulted in DMCA takedowns of datasets such as the Pile. Terms can also change: Ai2 switched Dolma to the ODC-By license in April 2024.10115

Web data includes personal information, and stewards document different responses. FineWeb replaces email addresses and public IP addresses, says some personal information is still likely to remain, and offers a removal form; Dolma's card also links a form for requesting removal of personal data. Website owners who do not want Common Crawl to crawl their sites can block its CCBot crawler in robots.txt.453