Evaluation toolEvaluation tool

Humanity's Last Exam (HLE)

Humanity's Last Exam (HLE) is an evaluation tool maintained by Center for AI Safety1 2 3 9 and Scale AI1 9, version HLE (2,500 questions, finalized April 2025); HLE-Rolling; HLE-Diamond (September 2026), released Jan 2025. Its code is public2 3 under the MIT License (evaluation code)3.

A multimodal benchmark of 2,500 expert-written, closed-ended academic questions across more than a hundred subjects, including mathematics, the humanities, and the natural sciences, in multiple-choice and short-answer formats suited to automated grading. It was organized by teams at the Center for AI Safety and Scale AI with questions from nearly 1,000 subject-matter contributors, and was published in Nature in January 2026.1 2

Website: Website (external site: lastexam.ai)
Repository: GitHub repository (external site: github.com)
Dataset hub: Dataset (Hugging Face, cais/hle) (external site: huggingface.co)
Paper: Paper (arXiv 2501.14249) (external site: arxiv.org)
Paper: Nature paper (external site: nature.com)
Release notes: HLE-Diamond release notes (external site: lastexam.ai)

On AI on Hugging Face: cais/hle, cais/hle-diamond

Last reviewedSep 29, 2026Entry updated Sep 29, 2026Documented release Jan 2025

Availability and license

Overall availability

Partial

The public question set is on Hugging Face behind an automatic access gate that asks users to share contact information; the evaluation code is public on GitHub. A private held-out question set is not released.4 5 2 1

Availability fact review: not separately recorded

Availability is separate from permission: read the license before using or redistributing.

MIT License (dataset) (external site: huggingface.co)4 5
Applies to: Data · SPDX MIT · on the rubric’s OSI-approved software-license list · fact-level rights review not recorded
Plain-language guide to the MIT License
MIT License (evaluation code) (external site: raw.githubusercontent.com)3
Applies to: Code · SPDX MIT · on the rubric’s OSI-approved software-license list · fact-level rights review not recorded
Plain-language guide to the MIT License

The dataset and repository include a canary string intended to help model developers keep the benchmark out of training data. The separate cais/hle-diamond dataset's metadata also lists the MIT license and uses the same automatic access gate.2 4 6

Component reuse rights

A readable or downloadable component is not automatically reusable. These indicators concern recorded license evidence, not system certification.

weights: Unknown — no complete fact-level rights review
code: Unknown — no complete fact-level rights review
data: Unknown — no complete fact-level rights review
documentation: Unknown — no complete fact-level rights review

No complete system-rights review is recorded for this release.

Public materials checklist

Items for a evaluation tool under USASI rubric v0.2. Unknown means unassessed or insufficient evidence.

Public materials checklist for Humanity's Last Exam (HLE)
Item	Status	Notes and evidence
CodeIs the evaluation code published?	Public	The hle_eval scripts generate model predictions through the openai-python interface and grade them with a judge model; MIT license.2 3
Tasks / dataAre the tasks or test data available?	Partial	The 2,500 public questions are downloadable after accepting the Hugging Face access prompt; a private held-out set is kept back to assess overfitting.4 5 1
MethodologyIs the method for scoring described?	Public	The paper (arXiv, with a Nature version) describes the benchmark, whose questions each have a known, unambiguous, verifiable answer that cannot be quickly found by internet search. The website explains that calibration error is measured by asking models for an answer and a 0-100% confidence, with answers graded by a judge model.8 1
ReproducibilityAre instructions for reproducing results published?	Public	The README gives commands for running predictions and the judge, notes temperature 0 as the default, and advises at least 8,192 completion tokens for reasoning models.2
LimitationsAre known limitations documented?	Public	The website states that HLE tests structured academic problems rather than open-ended research or creative problem solving, and that high accuracy alone would not indicate autonomous research ability. Changes to the question set (removals, re-additions, updates, and additions) are listed in the public HLE-Rolling change log, and HLE-Diamond is described as a subset refined over a year of cleaning.1 10 9

What it is useful for

Measuring language model accuracy and calibration on difficult closed-ended academic questions; the organizers keep a private held-out question set to check for overfitting.1 2

Run and use notes

Documented facts only. No hardware or performance claims are made without a cited source and stated assumptions.

HLE-Rolling (October 2025) is a dynamic fork updated through a public change log, and HLE-Diamond (September 22, 2026) is a refined 1,000-question subset split evenly between reasoning and knowledge questions, published as cais/hle-diamond.1 9 10

Organization context

In the news

Dated, sourced updates in this catalog's news that mention this entry.

Sep 22, 2026 · Release
CAIS and Scale AI release HLE-Diamond, a refined 1,000-question subset of Humanity's Last Exam
On September 22, 2026, the Center for AI Safety and Scale AI released HLE-Diamond, which the release notes describe as a refined subset of the Humanity's Last Exam (HLE) question collection, produced through a year-long process of cleaning and refinement with input from research communities. HLE-Diamond has 1,000 questions, split evenly into 500 reasoning and 500 knowledge questions, and is designed to be answered closed-book, with separate recommended settings for evaluations that use tools. The dataset is on Hugging Face as cais/hle-diamond, where the card lists an MIT license, requires agreeing to share contact information before access, and asks that the dataset not be publicly shared or re-uploaded.

U.S. eligibility

Project eligibility rests on documented governing or maintaining entities, not on contributors.

Eligible · basis: U.S.-governed project

The benchmark's organizing team is drawn from the Center for AI Safety and Scale AI; its GitHub repository is in the centerforaisafety organization under an MIT license held by centerforaisafety, and its dataset is published under CAIS's Hugging Face organization. The Center for AI Safety is a San Francisco 501(c)(3) nonprofit listed by the IRS. Scale AI is covered by its own catalog record.1 2 3 7 11 12

Assessed Sep 29, 2026

Sources

1.
Humanity's Last Exam (external site: lastexam.ai)
Center for AI Safety and Scale AI · Official page · accessed Sep 29, 2026
2.
centerforaisafety/hle README (external site: github.com)
Center for AI Safety · Repository · accessed Sep 29, 2026
3.
centerforaisafety/hle LICENSE (external site: raw.githubusercontent.com)
Center for AI Safety · License · accessed Sep 29, 2026
4.
cais/hle dataset card (external site: huggingface.co)
Center for AI Safety · Dataset card · accessed Sep 29, 2026
5.
cais/hle (Hugging Face dataset API metadata) (external site: huggingface.co)
Hugging Face · Other · accessed Sep 29, 2026
6.
cais/hle-diamond (Hugging Face dataset API metadata) (external site: huggingface.co)
Hugging Face · Other · accessed Sep 29, 2026
7.
cais organization overview (Hugging Face API) (external site: huggingface.co)
Hugging Face · Other · accessed Sep 29, 2026
8.
Humanity's Last Exam (arXiv 2501.14249) (external site: arxiv.org)
arXiv · Paper · published Jan 24, 2025 · accessed Sep 29, 2026
9.
HLE-Diamond release notes (external site: lastexam.ai)
Center for AI Safety and Scale AI · Release notes · published Sep 22, 2026 · accessed Sep 29, 2026
10.
hle-rolling-changes.txt (centerforaisafety/hle) (external site: github.com)
Center for AI Safety · Release notes · accessed Sep 29, 2026
11.
Donate | CAIS (external site: safe.ai)
Center for AI Safety · Official page · accessed Sep 29, 2026
12.
Exempt Organizations Business Master File Extract - California (eo_ca.csv) (external site: irs.gov)
Internal Revenue Service · Filing · accessed Sep 29, 2026

Email a correction Report a correction Edit this entry

This listing is not an endorsement, a safety assessment, or a federal approval.

Cite this page

APA style

USASI. (2026, September 29). Humanity's Last Exam (HLE). United States of America Superintelligence. https://unitedstatesofamericasuperintelligence.com/open/humanitys-last-exam/

BibTeX

@misc{usasi_humanitys_last_exam,
  author = {{USASI}},
  title = {Humanity's Last Exam (HLE)},
  year = {2026},
  month = sep,
  howpublished = {\url{https://unitedstatesofamericasuperintelligence.com/open/humanitys-last-exam/}},
  note = {United States of America Superintelligence. Last updated 2026-09-29}
}

The date is when this page was last updated. Add the date you read it if your style needs one.

For maintainers: a “Listed in USASI” badge

Links to this page. It says the entry is listed, not endorsed or rated. About the badge

Markdown (for a README)

[![Listed in the USASI catalog](https://unitedstatesofamericasuperintelligence.com/badges/listed.svg)](https://unitedstatesofamericasuperintelligence.com/open/humanitys-last-exam/)

HTML

<a href="https://unitedstatesofamericasuperintelligence.com/open/humanitys-last-exam/"><img src="https://unitedstatesofamericasuperintelligence.com/badges/listed.svg" alt="Listed in the USASI catalog" height="20"></a>