Common Crawl corpus
Common Crawl corpus is a dataset from Common Crawl Foundation12, version CC-MAIN-2026-39 (latest crawl at review), released Sep 2026. The data is publicly available34 under the Common Crawl Terms of Use (limited, non-exclusive license)2.
An archive of web crawl data collected by Common Crawl's crawlers since 2008, released as periodic crawls. Each crawl is published as WARC files (raw HTTP responses, requests, and metadata), WAT files (computed metadata as JSON), and WET files (extracted plain text), together with URL indexes.137
- Documentation: Get started (data access) (external site: commoncrawl.org)
- License: Terms of Use (external site: commoncrawl.org)
- Documentation: FAQ (external site: commoncrawl.org)
- Dataset hub: Crawl index collections (collinfo.json) (external site: index.commoncrawl.org)
Availability and license
Overall availability
Downloadable over HTTPS from data.commoncrawl.org without an AWS account; access to the s3://commoncrawl bucket (AWS us-east-1) through the S3 API is limited to authenticated AWS users. Use is subject to Common Crawl's Terms of Use, and the crawled content may be subject to its original publishers' terms and copyrights.342
Availability is separate from permission: read the license before using or redistributing.
The Terms of Use (last updated March 7, 2024) grant a limited, non-transferable, non-sublicensable license to the service, which is defined to include the crawled content. They state that crawled content is the responsibility of the party it originated from, that it may be subject to separate terms, and that users must respect third parties' copyrights. Users must indemnify Common Crawl against third-party claims arising from their use of the service or crawled content, expressly including use in connection with AI and machine learning systems such as large language models. Disputes go to binding arbitration under California law.2
Component reuse rights
- weights
- Unknown — no complete fact-level rights review
- code
- Unknown — no complete fact-level rights review
- data
- Unknown — no complete fact-level rights review
- documentation
- Unknown — no complete fact-level rights review
No complete system-rights review is recorded for this release.
Public materials checklist
| Item | Status | Notes and evidence |
|---|---|---|
| AccessCan the data be obtained, and on what terms? | Public | Downloadable over HTTPS without an AWS account; S3 API access requires an authenticated AWS account.34 |
| ProvenanceAre the data's origins documented? | Public | WARC files keep the HTTP request, response, and crawl metadata for each captured page, and the crawler identifies itself as CCBot and checks robots.txt before fetching.356 |
| DocumentationIs there a datasheet, card, or equivalent documentation? | Public | The Get Started page and FAQ document the formats, indexes, and access methods.36 |
| LicensingAre the licensing terms stated? | Partial | The Terms of Use are published, but no open data license covers the crawled content itself; the Terms say that content may be subject to its owners' separate terms and require users to respect third parties' copyrights.2 |
| Stated limitationsDoes the documentation state known limitations or risks? | Public | The FAQ describes the dataset as a sample of the web that generally archives a randomly selected subset of a site rather than the whole site, and the Terms of Use state that Common Crawl cannot guarantee the accuracy, quality, or lawfulness of the crawled content.62 |
What it is useful for
Organization context
Provenance and derivatives
Catalog records that name this entry in their provenance:
In the news
Common Crawl releases experimental web graph embeddings for web hosts
On September 7, 2026, Common Crawl published an experimental dataset of 128-dimensional embeddings for web hosts, learned from hyperlinks alone in its host-level web graph for the November 2025, December 2025, and January 2026 crawls. The model uses no page text, metadata, or hostnames and was trained only on hosts with at least eight links. The data is published as Parquet files on Common Crawl's S3 bucket and on Hugging Face, where the dataset card says its use is subject to the Common Crawl Terms of Use; the source code is on GitHub. Common Crawl says it is yet to be seen whether it will publish graph embeddings on a regular basis.
U.S. eligibility
Sources
This listing is not an endorsement, a safety assessment, or a federal approval.
Share or cite this page
- X (opens X)
- Bluesky (opens Bluesky)
- LinkedIn (opens LinkedIn)
- Reddit (opens Reddit)
- Hacker News (opens Hacker News)
- Email (opens your email app)
- RSS updates
Cite this page
APA style
USASI. (2026, September 29). Common Crawl corpus. United States of America Superintelligence. https://unitedstatesofamericasuperintelligence.com/open/common-crawl-corpus/
BibTeX
@misc{usasi_common_crawl_corpus,
author = {{USASI}},
title = {Common Crawl corpus},
year = {2026},
month = sep,
howpublished = {\url{https://unitedstatesofamericasuperintelligence.com/open/common-crawl-corpus/}},
note = {United States of America Superintelligence. Last updated 2026-09-29}
}The date is when this page was last updated. Add the date you read it if your style needs one.
For maintainers: a “Listed in USASI” badge
Links to this page. It says the entry is listed, not endorsed or rated. About the badge
Markdown (for a README)
[](https://unitedstatesofamericasuperintelligence.com/open/common-crawl-corpus/)
HTML
<a href="https://unitedstatesofamericasuperintelligence.com/open/common-crawl-corpus/"><img src="https://unitedstatesofamericasuperintelligence.com/badges/listed.svg" alt="Listed in the USASI catalog" height="20"></a>