Research
Ai2 releases BenchMIRT, a method, code, and data for auditing LLM benchmarks prompt by prompt
Ai2 introduced BenchMIRT on September 1, 2026, describing it as a method for auditing LLM benchmarks at the level of individual prompts. According to Ai2, it uses multidimensional Item Response Theory to estimate which underlying capabilities are most closely associated with getting each question right. Ai2 published a technical report, code on GitHub, and data in a Hugging Face collection. The post does not state a license. USASI has not tested this method.1
Related catalog entries
Sources
Share or cite this page
- X (opens X)
- Bluesky (opens Bluesky)
- LinkedIn (opens LinkedIn)
- Reddit (opens Reddit)
- Hacker News (opens Hacker News)
- Email (opens your email app)
- RSS updates
Cite this page
APA style
USASI. (2026, October 13). Ai2 releases BenchMIRT, a method, code, and data for auditing LLM benchmarks prompt by prompt. United States of America Superintelligence. https://unitedstatesofamericasuperintelligence.com/news/ai2-releases-benchmirt/
BibTeX
@misc{usasi_news_ai2_releases_benchmirt,
author = {{USASI}},
title = {Ai2 releases BenchMIRT, a method, code, and data for auditing LLM benchmarks prompt by prompt},
year = {2026},
month = oct,
howpublished = {\url{https://unitedstatesofamericasuperintelligence.com/news/ai2-releases-benchmirt/}},
note = {United States of America Superintelligence. Last updated 2026-10-13}
}The date is when this page was last updated. Add the date you read it if your style needs one.