Benchmarks and evaluation

Evaluation records in this catalog are of three kinds. Harnesses run many tests through one interface: EleutherAI's lm-evaluation-harness tests generative language models on a large number of tasks, using local models or commercial APIs, and Stanford CRFM's HELM offers benchmarks in a standardized format, models from several providers behind one interface, and metrics beyond accuracy such as efficiency, bias, and toxicity. Benchmarks are the task sets themselves, such as Humanity's Last Exam, a set of multiple-choice and short-answer questions suited to automated grading. System benchmarks such as MLCommons' MLPerf measure how fast computer systems train a model to a target quality or process inputs with a trained model.1246

Who produced a result matters. Under MLPerf's submission rules, each submitter must review at least one other submission, and the submitted source code must be sufficient to reproduce the results. ARC Prize Foundation tests models for its Verified Leaderboard on a semi-private evaluation set, working with model providers it selects, while Community Leaderboard entries get a light review and are not verified by default.78

Benchmarks change over time, so a result belongs to a version. Terminal-Bench uses semantic versioning: a change that significantly alters the agent's environment is a major version that requires re-running agents, while a change to the verifier alone is minor and lets saved runs be re-graded, and each leaderboard corresponds to a specific dataset version. Humanity's Last Exam keeps a dated public log of questions added, removed, updated, and re-added. HELM entered maintenance mode on June 1, 2026: no new evaluations will be added to its leaderboards, and its maintainers warn that scenarios and models that depend on external APIs may break as those APIs change or providers deprecate models.953

How a model is reached limits what can be measured. In lm-evaluation-harness, models that do not return log probabilities, such as those behind chat-completion APIs, can run only generation tasks, while local models can run every task type; its README suggests checking a few sample outputs to confirm that answers are extracted and scored as intended. Test material can also reach training data. Humanity's Last Exam includes a canary string so model builders can filter it out, and ARC Prize Foundation acknowledges possible limited leakage over time because its tasks are sent to external APIs.148

This catalog does not publish the scores that benchmarks produce. Each evaluation record instead checks whether the evaluation code and the tasks or test data are available, whether the scoring method is described, whether instructions for reproducing results are published, and whether known limitations are documented.10