Model family

Pythia

Family overview — summarizes releases; licenses and availability belong to each release.

Pythia is EleutherAI's suite of language models built for research on how models learn during training and change with scale. The main suite has eight sizes from 70M to 12B parameters, each trained once on the Pile and once on a deduplicated Pile, on the same data in the same order, with 154 checkpoints per model. The current suite was retrained and re-released in April 2023; 14M and 31M models and additional random-seed runs were added later.1 2 3

Repository: Pythia repository (external site: github.com)
Paper: Pythia paper (arXiv 2304.01373) (external site: arxiv.org)
Model hub: EleutherAI on Hugging Face (external site: huggingface.co)

Last reviewedSep 29, 2026Entry updated Sep 29, 2026Documented release Apr 3, 2023

Releases assessed

Each release is assessed on its own. This family has no family-wide openness label.

Releases in the Pythia family
Release	Weights	Tier (USASI rubric v0.1)	License	Released
Pythia 12B	Weights: Public	Model-disclosure tier (USASI rubric v0.1): Fully open	Apache-2.0, Apache-2.0	Apr 3, 2023

What it is useful for

EleutherAI states the suite is intended for research on the behavior and limitations of language models, including interpretability and learning dynamics, and not for deployment.2 1

Organization context

EleutherAI

Provenance and derivatives

Trained from scratch by EleutherAI with its GPT-NeoX library on the Pile (standard models) or the globally deduplicated Pile (deduped models), about 300 billion tokens per model.1 2

Derived from: The Pile — Pretraining data for all Pythia models.

U.S. eligibility

Project eligibility rests on documented governing or maintaining entities, not on contributors.

Eligible · basis: U.S. nonprofit or lab

The models are developed and released by EleutherAI, which states it incorporated as a non-profit research institute in early 2023; the IRS exempt-organization extract for the District of Columbia lists EleutherAI Institute in Washington, D.C., as a 501(c)(3) organization. Eligibility of each release is assessed on the release record.2 4 5

Assessed Sep 29, 2026

Sources

1.
EleutherAI/pythia: Interpreting Transformers Across Time and Scale (external site: github.com)
EleutherAI · Repository · accessed Sep 29, 2026
2.
EleutherAI/pythia-12b model card (external site: huggingface.co)
EleutherAI · Model card · accessed Sep 29, 2026
3.
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (arXiv 2304.01373) (external site: arxiv.org)
EleutherAI (arXiv) · Paper · published Apr 3, 2023 · accessed Sep 29, 2026
4.
About | EleutherAI (external site: eleuther.ai)
EleutherAI · Official page · accessed Sep 29, 2026
5.
IRS Exempt Organizations Business Master File Extract (District of Columbia) (external site: irs.gov)
U.S. Internal Revenue Service · Filing · accessed Sep 29, 2026

Report a correction Edit this entry

This listing is not an endorsement, a safety assessment, or a federal approval.