Evaluation toolEvaluation tool

OpenAI Evals

OpenAI Evals is an evaluation tool maintained by OpenAI1 3, released Mar 2023. Its code is public1 3 under the MIT License3.

OpenAI Evals is an open-source framework from OpenAI for evaluating language models and systems built on them, together with a registry of existing evals. Evals are defined in YAML and run from the oaieval and oaievalset command-line tools against "completion functions", which by default call models through the OpenAI API. The repository is not archived, but activity since mid-2024 has been limited to maintenance, and its README now points users to OpenAI's hosted Evals in the OpenAI dashboard.2 4 6 7

Repository: GitHub repository (external site: github.com)
Documentation: How to run evals (docs/run-evals.md) (external site: github.com)
Documentation: Eval templates (docs/eval-templates.md) (external site: github.com)
License: License (MIT, with per-dataset licenses) (external site: raw.githubusercontent.com)

Last reviewedSep 29, 2026Entry updated Sep 29, 2026Documented release Mar 2023

Availability and license

Overall availability

Public

Source code on GitHub under the MIT License and installable from PyPI as evals. The eval registry data is stored with Git LFS and must be fetched separately. Running evals against OpenAI models requires an OpenAI API key and incurs API costs.2 3 7

Availability fact review: not separately recorded

Availability is separate from permission: read the license before using or redistributing.

MIT License (external site: raw.githubusercontent.com)3
Applies to: Code · SPDX MIT · on the rubric’s OSI-approved software-license list · fact-level rights review not recorded
Plain-language guide to the MIT License

The license file states that MIT applies to everything except the datasets it lists separately, which keep their own licenses (for example ODC-By, CC0, CC BY, CC BY-SA, and, for some components of the steganography and theory-of-mind evals, CC BY-NC 4.0). The README states that contributors agree to license their eval logic and data under MIT and that OpenAI may use contributed data in future service improvements.3 2

Component reuse rights

A readable or downloadable component is not automatically reusable. These indicators concern recorded license evidence, not system certification.

weights: Unknown — no complete fact-level rights review
code: Unknown — no complete fact-level rights review
data: Unknown — no complete fact-level rights review
documentation: Unknown — no complete fact-level rights review

No complete system-rights review is recorded for this release.

Public materials checklist

Items for a evaluation tool under USASI rubric v0.2. Unknown means unassessed or insufficient evidence.

Public materials checklist for OpenAI Evals
Item	Status	Notes and evidence
CodeIs the evaluation code published?	Public	Published on GitHub under the MIT License.1 3
Tasks / dataAre the tasks or test data available?	Public	Eval definitions are YAML files under evals/registry/evals; the data files under evals/registry/data are stored with Git LFS. Some datasets carry their own licenses.2 4 3
MethodologyIs the method for scoring described?	Public	docs/eval-templates.md specifies the matching rules of the basic templates and the parameters of the model-graded classification template.5
ReproducibilityAre instructions for reproducing results published?	Public	docs/run-evals.md documents the oaieval and oaievalset commands, threading and timeout settings, and local JSONL logging.4
LimitationsAre known limitations documented?	Partial	The documentation notes operational limits (a single eval cannot be resumed mid-run; runs sometimes hang after the final report), and the README says evals with custom code are not being accepted. A general discussion of eval validity was not found in the pages read.4 2

What it is useful for

Writing evals from templates (exact, includes, fuzzy, and JSON match, or model-graded classification) without custom code, running them against OpenAI API models or other completion functions, and logging results locally.5 4 2

Run and use notes

Documented facts only. No hardware or performance claims are made without a cited source and stated assumptions.

The README gives Python 3.9 as the minimum version. The latest release on PyPI is 3.0.1.post1, published May 1, 2024.2 7
Recent commits are maintenance: a December 2024 README change linking to the hosted Evals product, removal of an eval suite with defunct dependencies in November 2025, and pinning of CI and pre-commit references in April 2026.6

Organization context

OpenAI

U.S. eligibility

Project eligibility rests on documented governing or maintaining entities, not on contributors.

Eligible · basis: U.S.-governed project

The project is published in OpenAI's GitHub organization, and its MIT license names OpenAI as copyright holder. OpenAI Group PBC lists its address as 1455 3rd Street, San Francisco, California, in a February 2026 agreement filed with the SEC.1 3 8

Assessed Sep 29, 2026

Sources

1.
openai/evals (external site: github.com)
OpenAI · Repository · accessed Sep 29, 2026
2.
openai/evals README.md (external site: raw.githubusercontent.com)
OpenAI · Documentation · accessed Sep 29, 2026
3.
openai/evals LICENSE.md (external site: raw.githubusercontent.com)
OpenAI · License · accessed Sep 29, 2026
4.
How to run evals (docs/run-evals.md) (external site: raw.githubusercontent.com)
OpenAI · Documentation · accessed Sep 29, 2026
5.
Existing templates for evals (docs/eval-templates.md) (external site: raw.githubusercontent.com)
OpenAI · Documentation · accessed Sep 29, 2026
6.
Commits · openai/evals (external site: github.com)
OpenAI · Repository · accessed Sep 29, 2026
7.
evals on PyPI (JSON metadata) (external site: pypi.org)
Python Package Index · Documentation · published May 1, 2024 · accessed Sep 29, 2026
8.
Exhibit 10.1: Equity commitment letter agreement between OpenAI Group PBC and Amazon (external site: sec.gov)
U.S. Securities and Exchange Commission (Amazon.com, Inc. filing) · Filing · published Feb 27, 2026 · accessed Sep 29, 2026

Email a correction Report a correction Edit this entry

This listing is not an endorsement, a safety assessment, or a federal approval.

Cite this page

APA style

USASI. (2026, September 29). OpenAI Evals. United States of America Superintelligence. https://unitedstatesofamericasuperintelligence.com/open/openai-evals/

BibTeX

@misc{usasi_openai_evals,
  author = {{USASI}},
  title = {OpenAI Evals},
  year = {2026},
  month = sep,
  howpublished = {\url{https://unitedstatesofamericasuperintelligence.com/open/openai-evals/}},
  note = {United States of America Superintelligence. Last updated 2026-09-29}
}

The date is when this page was last updated. Add the date you read it if your style needs one.

For maintainers: a “Listed in USASI” badge

Links to this page. It says the entry is listed, not endorsed or rated. About the badge

Markdown (for a README)

[![Listed in the USASI catalog](https://unitedstatesofamericasuperintelligence.com/badges/listed.svg)](https://unitedstatesofamericasuperintelligence.com/open/openai-evals/)

HTML

<a href="https://unitedstatesofamericasuperintelligence.com/open/openai-evals/"><img src="https://unitedstatesofamericasuperintelligence.com/badges/listed.svg" alt="Listed in the USASI catalog" height="20"></a>