AILuminate
Version Safety v1.0 and v1.1; Jailbreak v0.5
Maintained by MLCommons (AI Risk & Reliability working group)138
AILuminate is MLCommons' family of safety and security benchmarks for generative AI systems, organized around 12 hazard categories. The v1.0 safety benchmark (December 2024) tests general-purpose chat systems with more than 24,000 human-written prompts and grades responses with an ensemble of evaluator models; a jailbreak benchmark measures how safety degrades under attack.1387
- Website: AILuminate overview (external site: mlcommons.org)
- Documentation: Safety methodology (external site: mlcommons.org)
- Repository: AILuminate v1.0 demo prompt set (external site: github.com)
- Repository: ModelBench benchmarking framework (external site: github.com)
- Paper: AILuminate v1.0 paper (arXiv 2503.05731) (external site: arxiv.org)
Availability and license
Overall availability
1,200-prompt demo sets in English and French (each a 10% sample of the corresponding 12,000-prompt practice set) are public under CC BY 4.0, and the ModelBench code is public under Apache 2.0. The full English and French practice prompt sets are offered to MLCommons members on request. The official test prompts are hidden, and official grading uses MLCommons' evaluator ensemble.861023
Availability is separate from permission: read the license before using or redistributing.
Creative Commons Attribution 4.0 International (demo prompt sets) (external site: creativecommons.org)8
Apache License 2.0 (ModelBench) (external site: raw.githubusercontent.com)1110
The ailuminate repository's README licenses the demo prompt data under CC BY 4.0, while the repository's LICENSE.md file contains the Apache License 2.0. The README warns that the prompts were written to elicit hazardous responses and may be offensive or disturbing.89
Public materials checklist
| Item | Status | Notes and evidence |
|---|---|---|
| CodeIs the evaluation code published? | Public | ModelBench (which now includes ModelGauge) runs tests against systems under test and aggregates hazard scores; Apache 2.0.1011 |
| Tasks / dataAre the tasks or test data available? | Partial | Only the 1,200-prompt English and French demo sets are public; full practice sets go to MLCommons members and the official test prompts are hidden to limit overfitting.862 |
| MethodologyIs the method for scoring described? | Public | The safety methodology page and the v1.0 paper describe the hazard taxonomy, personas, practice and official prompt splits, the evaluator ensemble, and grading against reference models on a five-tier scale.27 |
| ReproducibilityAre instructions for reproducing results published? | Partial | ModelBench documents how to run a practice benchmark (using a Llama Guard evaluator via Together AI); official results depend on the private prompt set and official evaluator ensemble.102 |
| LimitationsAre known limitations documented? | Public | The v1.0 paper names evaluator uncertainty and the single-turn format as limitations and lists multi-turn, multimodal, additional-language, and new-hazard coverage as needed work.7 |
What it is useful for
Run and use notes
- The AILuminate page lists safety benchmarks for text in English, French, and Chinese, and jailbreak benchmarks for text and text-plus-image inputs in English; an agentic workstream is in development.1
- In February 2026 MLCommons published a jailbreak taxonomy methodology paper, which it described as groundwork rather than a new benchmark release.12
- MLCommons announced AILuminate v1.1 in February 2025, adding French; the announcement calls the English benchmark v1.0 and the French one v1.1. The Safety page still labels both the English and French official results as v1.0.54
Organization context
U.S. eligibility
Eligible · basis: U.S.-governed project
AILuminate is developed by the MLCommons AI Risk & Reliability working group and published by MLCommons, which licenses the demo prompt data and hosts the ModelBench code. MLCommons Association is listed by the IRS as a 501(c)(6) organization with a Dover, Delaware address.381013
Sources
This listing is not an endorsement, a safety assessment, or a federal approval.