DatasetDataset

Essential-Web v1.0

Essential-Web v1.0 is a dataset from Essential AI1 2, version v1.0, released Jun 2025. The data is publicly available1 under the Open Data Commons Attribution License v1.01.

Essential-Web v1.0 is a web text dataset of about 24 trillion tokens in 23.6 billion documents, built by Essential AI from 101 Common Crawl snapshots. Every document carries metadata from a 12-category taxonomy covering subject (Free Decimal Correspondence, a Dewey Decimal-inspired scheme), page type, reasoning depth, education level, and quality, so subsets can be selected with metadata filters.1 2

Dataset hub: Dataset card (Hugging Face) (external site: huggingface.co)
Paper: Essential-Web v1.0 paper (arXiv 2506.14111) (external site: arxiv.org)
Model hub: EAI-Distill-0.5b labeling model (external site: huggingface.co)
Repository: eai-taxonomy code (external site: github.com)

On AI on Hugging Face: EssentialAI/eai-distill-0.5b, EssentialAI/essential-web-v1.0

Last reviewedSep 29, 2026Entry updated Sep 29, 2026Documented release Jun 2025

Availability and license

Overall availability

Public

Downloadable from Hugging Face without gating. The card asks users to also follow the Common Crawl terms of use.1

Availability fact review: not separately recorded

Availability is separate from permission: read the license before using or redistributing.

Open Data Commons Attribution License v1.0 (external site: opendatacommons.org)1
Applies to: Data · SPDX ODC-By-1.0 · fact-level rights review not recorded
Plain-language guide to the Open Data Commons Attribution License v1.0

The card says Essential AI's contributions are released under ODC-By, that users should also follow the Common Crawl terms of use, and that Essential AI does not alter the license of any underlying data.1

Component reuse rights

A readable or downloadable component is not automatically reusable. These indicators concern recorded license evidence, not system certification.

weights: Unknown — no complete fact-level rights review
code: Unknown — no complete fact-level rights review
data: Unknown — no complete fact-level rights review
documentation: Unknown — no complete fact-level rights review

No complete system-rights review is recorded for this release.

Public materials checklist

Items for a dataset under USASI rubric v0.2. Unknown means unassessed or insufficient evidence.

Public materials checklist for Essential-Web v1.0
Item	Status	Notes and evidence
AccessCan the data be obtained, and on what terms?	Public	1
ProvenanceAre the data's origins documented?	Public	The card lists the Common Crawl snapshot ranges and the processing steps: global and MinHash deduplication, quality signals from a RedPajama-Data-V2-style pipeline and the DCLM-baseline fastText classifier, filtering, and taxonomy labeling.1
DocumentationIs there a datasheet, card, or equivalent documentation?	Public	The card documents the schema and taxonomy codes; the paper describes the method.1 2
LicensingAre the licensing terms stated?	Public	ODC-By for Essential AI's contributions, alongside Common Crawl's terms of use.1
Stated limitationsDoes the documentation state known limitations or risks?	Unknown	The paper has no dedicated limitations section; it notes that its small labeling model does worse than its teacher on some categories, such as extraction artifacts and education level.2

What it is useful for

Building domain-specific pretraining sets without custom classifiers by filtering on the taxonomy labels. Essential AI has published math, code, STEM, and medical subsets curated this way.1

Organization context

Essential AI

Provenance and derivatives

Built from Common Crawl: 89 snapshots from the DCLM pool (CC-MAIN-2013-20 to CC-MAIN-2022-49) plus 12 later snapshots (CC-MAIN-2023-06 to CC-MAIN-2024-38), all extracted with resiliparse. Taxonomy labels were produced by EAI-Distill-0.5b, which Essential AI fine-tuned from Qwen2.5-0.5B-Instruct (from Alibaba Cloud's Qwen family) on labels generated by Qwen2.5-32B-Instruct; the dataset card refers to the labeler as EAI-Taxonomy-0.5b.1 2 3 5

Derived from: Common Crawl — Source of all documents (101 snapshots).
Derived from: EAI-Distill-0.5b (external site: huggingface.co) — Essential AI labeling model fine-tuned from Qwen2.5-0.5B-Instruct.

What this catalog does not know

Unknown means the sources reviewed for this record do not document it. It is not evidence that something does not exist.

Stated limitations: unknown.

Have a primary source? How to report a correction.

U.S. eligibility

Project eligibility rests on documented governing or maintaining entities, not on contributors.

Eligible · basis: U.S.-governed project

Created and published by Essential AI under its Hugging Face organization; the paper lists the authors' affiliation as Essential AI, San Francisco, CA, and Essential AI's website gives the same location.1 2 4

Assessed Sep 29, 2026

Sources

1.
EssentialAI/essential-web-v1.0 dataset card (external site: huggingface.co)
Essential AI · Dataset card · accessed Sep 29, 2026
2.
Essential-Web v1.0: 24T tokens of organized web data (arXiv 2506.14111v2) (external site: arxiv.org)
Essential AI · Paper · published Jun 19, 2025 · accessed Sep 29, 2026
3.
EssentialAI/eai-distill-0.5b model card (external site: huggingface.co)
Essential AI · Model card · accessed Sep 29, 2026
4.
About | Essential AI (external site: essential.ai)
Essential AI · Official page · accessed Sep 29, 2026
5.
Qwen on Hugging Face (external site: huggingface.co)
Qwen (Alibaba Cloud) · Repository · accessed Sep 29, 2026

Email a correction Report a correction Edit this entry

This listing is not an endorsement, a safety assessment, or a federal approval.

Cite this page

APA style

USASI. (2026, September 29). Essential-Web v1.0. United States of America Superintelligence. https://unitedstatesofamericasuperintelligence.com/open/essential-web/

BibTeX

@misc{usasi_essential_web,
  author = {{USASI}},
  title = {Essential-Web v1.0},
  year = {2026},
  month = sep,
  howpublished = {\url{https://unitedstatesofamericasuperintelligence.com/open/essential-web/}},
  note = {United States of America Superintelligence. Last updated 2026-09-29}
}

The date is when this page was last updated. Add the date you read it if your style needs one.

For maintainers: a “Listed in USASI” badge

Links to this page. It says the entry is listed, not endorsed or rated. About the badge

Markdown (for a README)

[![Listed in the USASI catalog](https://unitedstatesofamericasuperintelligence.com/badges/listed.svg)](https://unitedstatesofamericasuperintelligence.com/open/essential-web/)

HTML

<a href="https://unitedstatesofamericasuperintelligence.com/open/essential-web/"><img src="https://unitedstatesofamericasuperintelligence.com/badges/listed.svg" alt="Listed in the USASI catalog" height="20"></a>