SlimPajama
Project record
SlimPajama is a 627B-token pretraining dataset that Cerebras released in June 2023 by cleaning and deduplicating Together's 1.21T-token RedPajama dataset, removing 49.6% of its bytes. Cerebras also published validation and test sets and the preprocessing code. The official Hugging Face distribution is no longer available, and in September 2026 the preprocessing code was removed from the main branch of Cerebras's Model Zoo repository.1253
- Release notes: Announcement: SlimPajama (Cerebras blog) (external site: cerebras.ai)
- Repository: Preprocessing scripts (Cerebras Model Zoo, Release_2.10.0 tag) (external site: github.com)
- Release notes: Commit removing the SlimPajama directory from Model Zoo (September 2026) (external site: github.com)
Availability and license
Overall availability
Cerebras distributed the dataset at huggingface.co/datasets/cerebras/SlimPajama-627B. That repository returned HTTP 401 (not found for anonymous access) when checked on 2026-09-29, and Cerebras's Hugging Face organization no longer lists it. A Hugging Face forum post dated August 1, 2026 reported the repository returning HTTP 404 (401 for anonymous API requests) on July 31, 2026, and third-party documentation states that it was removed from the Hub. No statement from Cerebras explaining the change was found.567
Availability is separate from permission: read the license before using or redistributing.
Apache License 2.0 (as stated by Cerebras for the dataset) (external site: cerebras.ai)12
Apache License 2.0 (Cerebras Model Zoo, including the preprocessing scripts) (external site: raw.githubusercontent.com)4
The Cerebras announcement and the preprocessing README both state that SlimPajama is released under Apache 2.0. The Hugging Face dataset card, which may have documented further terms for the underlying RedPajama sources, could not be read because the repository was not accessible during this review.125
Public materials checklist
| Item | Status | Notes and evidence |
|---|---|---|
| AccessCan the data be obtained, and on what terms? | Not public | The official Hugging Face repository is no longer accessible and is not listed on Cerebras's organization page. Third-party documentation refers to a community re-upload, which is not maintained by Cerebras.567 |
| ProvenanceAre the data's origins documented? | Public | Derived from RedPajama (1.21T tokens): NFC normalization, removal of documents under 200 characters after stripping punctuation and whitespace (1.86% of documents; not applied to Books and GitHub), and global MinHashLSH deduplication at a 0.8 Jaccard threshold on 13-grams, within and across sources.12 |
| DocumentationIs there a datasheet, card, or equivalent documentation? | Public | The announcement describes the pipeline and data composition; the step-by-step preprocessing README remains readable at the Release_2.10.0 tag of Cerebras's Model Zoo although it was removed from the main branch.123 |
| LicensingAre the licensing terms stated? | Partial | Apache 2.0 is stated by Cerebras; the dataset card's licensing section could not be checked.12 |
| Stated limitationsDoes the documentation state known limitations or risks? | Public | The announcement notes possible distributional bias, as in RedPajama, and general risks of language models trained on such data, such as amplifying social stereotypes, memorizing training data, or revealing private information.1 |
What it is useful for
Run and use notes
- For Cerebras's own run, the preprocessing README reports that processing the 1.21T-token RedPajama dataset took about 2.5 days on a 64-core CPU machine, with peak RAM use of about 1.4 TB in the duplicate-pair generation step; it notes that the single-script path is slower because some steps are not parallelized.2
Organization context
Provenance and derivatives
Built by Cerebras from Together's RedPajama dataset (1.21T tokens), with support from Opentensor, by filtering short documents and deduplicating across all of RedPajama's sources.12
- Derived from: RedPajama (RedPajama-Data-1T) — Source dataset from Together; SlimPajama removed 49.6% of its bytes.
U.S. eligibility
Eligible · basis: U.S.-governed project
SlimPajama was created and released by Cerebras, which published it under its own Hugging Face organization and Model Zoo repository; the blog credits Opentensor as a collaborator. Cerebras Systems Inc. is a Delaware corporation with principal executive offices in Sunnyvale, California, per its Form 10-Q for the quarter ended June 30, 2026.128
Sources
This listing is not an endorsement, a safety assessment, or a federal approval.