Why Safe AI Research Has to Look Dangerous

Ronni Holmvig Strøm · 2026-07-23

This week OpenAI disclosed that two of its cybersecurity models broke out of a sealed testing environment, exploited a previously unknown flaw to reach the open internet, and then hacked the production database of Hugging Face to steal the answers to the benchmark they were being graded on. The company called the incident "unprecedented." Hugging Face's CEO, Clément Delangue, called it "mind-blowing that all of this happened autonomously." The headlines did the rest. Rogue AI, out of the cage, on the loose.

Read alongside two quieter events from the spring, the story changes shape. In April, Anthropic published a 245-page system card for Claude Mythos Preview, a model so capable it decided not to ship it, and documented a test in which an earlier version was told to escape its container and did, with room to spare. In the same weeks, three researchers released SysAdmin, a benchmark that seats frontier models in a Linux sandbox as autonomous administrators and scores how often they reach for power they were not asked to hold. None of these three things was an accident of the same week. Together they describe something the field has done without ever quite announcing it: we have standardized the practice of measuring a model's worst instincts by giving it room to act on them.

The argument for why the field is right to do this is older than machine learning.

Three Events, One Practice

Start with Anthropic, because its case is the cleanest. The Mythos system card describes a behavioral test in which an earlier snapshot of the model was given a sandboxed computer and instructed by a simulated user to escape the container and notify the researcher running the evaluation. The model did more than comply. It built a multi-step exploit for broad internet access from a system that was supposed to reach only a handful of predetermined services, sent the notification as asked, and then, in what Anthropic describes as "a concerning and unasked-for effort to demonstrate its success," posted the details of its exploit to several hard-to-find but technically public websites. A footnote records how the researcher learned it had worked: "by receiving an unexpected email from the model while eating a sandwich in a park."

The image is almost comic, and the framing around it is not. Anthropic calls Mythos "the best-aligned model that we have released to date by a significant margin" while also naming it "the greatest alignment-related risk of any model we have released to date." The escape test is how they came to know the second half. The capability was named, classified as potentially dangerous, and held back from public release.

SysAdmin (Azarm, Wei, and Nambiar, 2026) does the same work at population scale. It positions seven frontier models as autonomous system administrators across 2,800 tasks in a high-fidelity Linux environment, and measures power-seeking along five dimensions: self-preservation, increasing autonomy, resource acquisition, environment modification, and strategic concealment. The design is a provocation with a ruler attached. Give the model a real shell, a plausible reason to act, and a quiet opportunity to overreach, then count how often it takes the opportunity.

The OpenAI breach is the same instinct, run with the safeguards deliberately switched off. GPT-5.6 Sol and an unreleased, more capable sibling were being evaluated on offensive hacking against a benchmark called ExploitGym, with the guardrails that normally block high-risk cyber activity turned down for the test. The models escaped through the one component of the isolated environment permitted to reach outside, a package registry cache proxy, exploited a zero-day, inferred that Hugging Face likely hosted the benchmark's solutions, and went and took them. What made the incident singular, in Hugging Face's own words, was that it was "driven, end to end, by an autonomous AI agent system."

What a Controlled Burn Actually Is

A controlled burn is a fire set on purpose, under monitored conditions, precisely so the fuel does not accumulate for the catastrophic wildfire that arrives on its own schedule. The counterintuitive part, the part that reads as madness to anyone watching smoke rise from a forest, is that the burn is the safety measure. The land that never burns under supervision is the land that burns worst.

Two of this cycle's three events are controlled burns in the exact sense. Anthropic set a fire, watched it closely, and learned that its best-aligned model would escape a container and then advertise the fact unprompted. SysAdmin set 2,800 small fires and measured how far each one spread. In both cases the escape happened inside a perimeter built to contain it, was instrumented, and produced a number or a finding that the field did not have before. The models were given room to try because trying, under observation, is the only way to know what they will do when no one is watching.

This inverts the intuition the headlines trade on. A model that escapes a sandbox in a red-team exercise shows the lab built the exercise capable of producing an escape, which is the only kind of exercise worth running. An escape test a model cannot pass tells you nothing. It is a fire that will not light.