Evaluation toolEvaluation tool

LM Evaluation Harness

Version 0.4.13

An open-source framework from EleutherAI for evaluating language models on many benchmark tasks through one interface. It supports local models (for example through Hugging Face Transformers or vLLM) and hosted model APIs, and its README lists more than 60 standard academic benchmarks with hundreds of subtasks and variants.1

Repository: GitHub repository (external site: github.com)
Release notes: Release v0.4.13 (external site: github.com)
Documentation: New task guide (external site: github.com)
License: License (MIT) (external site: raw.githubusercontent.com)

Last reviewedSep 29, 2026Entry updated Sep 29, 2026Documented release Aug 31, 2026

Availability and license

Overall availability

Public

Source code on GitHub under the MIT License; installable from PyPI as lm-eval, with model backends installed as optional extras.1 5 2

Availability is separate from permission: read the license before using or redistributing.

MIT License (external site: raw.githubusercontent.com)2
Applies to: Code · SPDX MIT · on the rubric’s OSI-approved list

The new-task guide states that task data is downloaded and managed through the Hugging Face datasets API, so benchmark data comes from separately published datasets rather than from the harness repository.3

Public materials checklist

Items for a evaluation tool under USASI rubric v0.1. Unknown means unassessed or insufficient evidence.

Public materials checklist for LM Evaluation Harness
Item	Status	Notes and evidence
CodeIs the evaluation code published?	Public	Published on GitHub under the MIT License.1 2
Tasks / dataAre the tasks or test data available?	Public	Task configurations are in the repository; task data is loaded from datasets on the Hugging Face Hub (or local files) as specified in each task's configuration.3 1
MethodologyIs the method for scoring described?	Public	The new-task guide describes generative and multiple-choice (log-likelihood) task types and how metrics and aggregations are declared; the README sets out the order of preference used to choose prompting and evaluation procedures.3 1
ReproducibilityAre instructions for reproducing results published?	Public	The README documents command-line usage, publicly available prompts, result caching, and logging; versioned releases are published on GitHub and PyPI.1 4 5
LimitationsAre known limitations documented?	Partial	The README documents operational limitations (no native multi-node evaluation, early-stage support for the PyTorch MPS backend, and request types some backends do not support). A general discussion of benchmark validity limits was not found in the pages read.1

What it is useful for

Running benchmark evaluations of language models with shared, publicly available prompts, and adding new tasks through YAML task configurations.1 3

Run and use notes

Documented facts only. No hardware or performance claims are made without a cited source and stated assumptions.

Since December 2025 the base package no longer bundles transformers or torch; backends are installed as extras such as lm_eval[hf], lm_eval[vllm], or lm_eval[api], per the README.1

Organization context

EleutherAI

U.S. eligibility

Project eligibility rests on documented governing or maintaining entities, not on contributors.

Eligible · basis: U.S.-governed project

The project is maintained in EleutherAI's GitHub organization, its MIT license names EleutherAI as copyright holder, and the PyPI package lists EleutherAI as author. EleutherAI Institute is listed by the IRS as a 501(c)(3) organization in Washington, D.C.1 2 5 6

Assessed Sep 29, 2026

Sources

1.
EleutherAI/lm-evaluation-harness (external site: github.com)
EleutherAI · Repository · accessed Sep 29, 2026
2.
lm-evaluation-harness LICENSE.md (external site: raw.githubusercontent.com)
EleutherAI · License · accessed Sep 29, 2026
3.
lm-evaluation-harness new task guide (docs/new_task_guide.md) (external site: github.com)
EleutherAI · Documentation · accessed Sep 29, 2026
4.
Release v0.4.13 · EleutherAI/lm-evaluation-harness (external site: github.com)
EleutherAI · Release notes · published Aug 31, 2026 · accessed Sep 29, 2026
5.
lm-eval on PyPI (external site: pypi.org)
Python Package Index · Documentation · published Aug 31, 2026 · accessed Sep 29, 2026
6.
Exempt Organizations Business Master File Extract - District of Columbia (eo_dc.csv) (external site: irs.gov)
Internal Revenue Service · Filing · accessed Sep 29, 2026

Report a correction Edit this entry

This listing is not an endorsement, a safety assessment, or a federal approval.