AI safety, security, and trust

NIST's AI Risk Management Framework, released on January 26, 2023, is intended for voluntary use, and its core has four functions: Govern, Map, Measure, and Manage. NIST says the framework is being revised as part of the White House AI Action Plan. NIST's Dioptra test platform supports the Measure function and lists red-teaming in a controlled environment among its uses. NIST's Center for Advancing Innovation and Standards for Super Intelligence (CAISSI) says it will help industry develop voluntary standards and lead unclassified evaluations focused on demonstrable risks such as cybersecurity, biosecurity, and chemical weapons.1234

Several developers publish frameworks for testing new models for risks of severe harm. OpenAI's Preparedness Framework (version 2, April 2025) tracks biological and chemical, cybersecurity, and AI self-improvement capabilities against set thresholds. Google DeepMind's Frontier Safety Framework uses Critical Capability Levels, thresholds at which a model may pose heightened risk of severe harm without mitigations. Anthropic's Responsible Scaling Policy is at version 3.4, effective July 8, 2026. Model cards report how one release was tested: OpenAI's gpt-oss card says that because open weights can be fine-tuned, OpenAI also tested adversarially fine-tuned versions of gpt-oss-120b, which it did not release.5678

Testing tools probe models for unwanted behavior. In Petri, an auditor model runs multi-turn scenarios with a target model and a judge model scores the transcripts. Anthropic says Petri has been part of its alignment assessment of every Claude model since Claude Sonnet 4.5, and in May 2026 it handed Petri's development to Meridian Labs, an AI evaluation nonprofit. METR's Inspect-Hawk runs evaluations written with Inspect AI, the UK AI Security Institute's open-source framework, in isolated Kubernetes pods. MLCommons' AILuminate benchmarks cover 12 hazard categories and include jailbreak tests with text and with text plus images.9101112

Safety classifiers screen what goes into and comes out of another model. Meta's Llama Guard 4 is a 12-billion-parameter classifier for text and images that labels a prompt or a response as safe or unsafe and, if unsafe, lists the violated categories from a taxonomy based on MLCommons' hazard categories. OpenAI's gpt-oss-safeguard models, built on gpt-oss, instead classify text against a written policy that the developer supplies. OpenAI says they are intended for safety use cases and work correctly only with its harmony response format.131415

Provenance records where content came from. NIST's report on synthetic content (NIST AI 100-4) covers watermarks, metadata, and detection, and says none of these techniques is a comprehensive solution on its own. The C2PA standard's Content Credentials are a cryptographically bound record of an asset's origin, edits, and use of AI; they show whether that record is intact and signed by a trusted party, not whether its claims are true. Google DeepMind has published watermarking code for generated text (SynthID Text, a reference implementation not meant for production) and for AI-generated biological sequences and structures (SynthID Bio).16171819

Prompt injection is a security risk for systems built on language models. NIST's taxonomy of attacks (NIST AI 100-2 E2025) calls it direct when the system's main user supplies instructions that are appended to higher-trust ones, such as the system prompt, and indirect when a third party plants instructions in data the system reads, such as content fetched by an agent or a retrieval system. NIST says current mitigations do not offer full protection, so designers may assume injection is possible whenever a model sees untrusted input. For agents that use tools, the added risks include running arbitrary code and exfiltrating data.20