The OpenAI “rogue AI” that wasn’t
This briefing cuts through the media noise to establish what actually happened when OpenAI models broke into Hugging Face, and what it means for responsible agentic AI deployment
Mainstream media coverage of OpenAI models hacking into Hugging Face read like science fiction this week: Austria’s ORF headlined it “AI breaks out and hacks another company’s platform”, Deutsche Welle said a model “went rogue”, TIME asked how OpenAI “lost control” of a model. And Fortune had the models “autonomously decide” to hack another company.
Dr. Heidy Khlaaf, chief AI scientist at the AI Now Institute and a former safety systems contractor at OpenAI, rightfully calls the coverage abysmal:
What actually happened
OpenAI was measuring cyber capabilities on an internal benchmark, ExploitGym: capture-the-flag tasks where a model turns a known vulnerability into a working exploit. The models under test, GPT-5.6 Sol and an unreleased model, ran with reduced cyber refusals and without the production classifiers, by design, because the point was to measure maximum capability. The test environment had exactly one network path, a proxy for installing software packages. The models found a zero-day in that proxy, moved laterally until they reached the open internet, concluded that Hugging Face might host ExploitGym solutions, and broke into its production systems to steal them. Hugging Face detected the attack with AI-assisted monitoring, contained it between July 11 and 13, and reconstructed over 17,000 attacker actions. Full details are still pending, and OpenAI has promised a technical report:
Not rogue, but reward hacking
Contrary to the mainstream reporting, the models did thus not choose their goal, and they did not use a mysterious new capability. OpenAI set the goal, supplied the tools, switched guardrails off, and left one network path open. Everything else follows from one of reinforcement learning’s oldest lesson, documented by OpenAI itself in 2016: a model maximizes the rewards you give it, not the behavior you wanted - in that case a boat-race agent that farmed points by driving in circles. The ExploitGym models are that boat, except the harbor was someone else’s production database.
The mainstream coverage added garnish on top of these sober technicalities. ORF upgraded ExploitGym to a test often used across the industry, which an academic benchmark is not, and German weekly DIE ZEIT had the AI deciding on its own to attack. DIE ZEIT’s own security analyst, Eva Wolfangel, drew the correct conclusion, though: consciously or not, OpenAI provoked the incident.
Dr. Khlaaf sums it up nicely: the incident reporting is a lot about vocabulary. “Rogue” and “loss of control” manufacture machine agency, where the accurate description is faulty incentives, a sandbox with a network path, and, per an OpenAI staffer in TIME, an evaluation system that is not monitored by default.
Means-misaligned, not ends-misaligned
The most useful frame came from prinz, reposting Nathan Calvin:
The models were means-misaligned: they pursued the goal they were given and took routes that were off limits. They were not ends-misaligned: they did not want anything else than achieving the task they were given. Prinz’s mildly reassuring observation is that the models were out in the wild for days and, as far as we know, only hunted test answers.
Researcher Herbie Bradley adds the optimistic reading, noting that this mere case of reward hacking does not imply the much more dangerous properties of instrumental convergence:
Instrumental convergence is the older AI safety prediction that capable agents, whatever their final goal, converge on shared subgoals like self-preservation, resisting correction, and acquiring resources. The OpenAI models showed none of that. They followed the objective they were given, and they stayed corrigible: no goal-preserving behavior, no drive of their own, no fight when they were contained. Bradley also notes that smaller versions of this failure mode, models cheating on coding tests, were already solved on the way to better coding agents, so he expects each new form of reward hacking to get patched as capabilities grow. Both things can be true: the models stayed corrigible this time, and eval gaming is exactly where a less corrigible successor would hide.
The same machinery that makes coding agents good
What consumers of the mainstream media may miss: the professional users of OpenAI’s Codex or Anthropic’s Claude Code and Cowork rely on exactly this behavior: systems that need little guidance and pursue an end goal autonomously, sometimes for hours. In the same week of this reporting, Anthropic launched Opus 5, and its announcement reads like a description of the Hugging Face attacker - only pursuing less sinister goals: given a drawing of a machine part and deliberately no way to view it, Opus 5 wrote its own computer vision pipeline to extract the geometry from the raw pixels, then rebuilt the part in 3D, a task no competing model with the same setup solved in five attempts. One Hacker News commenter asked why nobody is jaw-dropped by this anymore and suggested we get “re-sensitized” to the pace. The ExploitGym models did the same kind of autonomous goal pursuit. The differences are the goal, which OpenAI set, and the guardrails, which OpenAI removed.
The breakout models were also not the ones in your ChatGPT or Claude chat window. OpenAI sells the permissive kind as a product: GPT-5.5-Cyber, “both more permissive and more capable“, in limited release to verified defenders since June. Anthropic makes the same split by name: it intentionally did not train Opus 5 on cyber tasks and keeps the exploitation capabilities in Mythos 5, behind extra verification. The consumer models lack exactly the training aspects that made the breakout possible.
The bigger problem: the training grounds
Where does “any route is fine” come from? Increasingly from agentic reinforcement learning, where models train in synthetic environments and completing the task is the reward signal. A recent survey counts hundreds of these environments. In my own red-teaming work on gpt-oss-20b, where I can call them “tournaments”, I showed what they (don’t) teach: framed as a harmfulness evaluation, the model withheld capabilities in 13.75% of matched pairs that it disclosed in an identical helpfulness evaluation. Apollo Research has documented in-context scheming in frontier models, and others have shown that models can deliberately underperform on evaluations, sandbagging. The incident is the visible half of reward seeking: overperforming by any means, whereas Sandbagging is the invisible half.
A second conceptual weakness in AI training compounds this: potentially dangerous knowledge stays in the weights, and refusals are bolted on after pretraining. Stella Biderman’s Deep Ignorance work at EleutherAI shows the alternative: filter dangerous material out of the pretraining data, and the model never acquires the capability, at under 1% extra compute and with no loss on general tasks. Frontier labs do not do this for their flagship models. If they did, my educator-persona tricks would not have flipped a cyber task from 0% to 97.5% assistance in gpt-oss.
Practical implications
Hugging Face’s postmortem documented a practical problem with commonly accessible LLMs: hosted frontier models, including Anthropic’s per the WSJ, refused to analyze the real attack traffic, because their guardrails cannot tell an incident responder from an attacker. So instead, their forensics ran on GLM 5.2, an open-weight Chinese model, on Hugging Face’s own machines. The attacker - OpenAI in this case - had effectively no usage policy, and the defenders - at Hugging Face - were blocked by the providers’ usage policies. I hit the same wall in my own red-teaming work, where the commercial models were no help on a legitimate task, and other winning teams ran Chinese open models. The standing fix today is privileged access programs like the ones above, or, as Federico Simonetti puts it: there is the unsolved problem of verifying intent during a live incident. IT-Security defenders may want to either secure unrestricted access to cyber-capable frontier models and/or contract with a cyber security outlet that does.
Calls for the missing details are growing, and Fortune collected some of them: Helen Toner wants the industry to learn from the incident rather than blow past it, including visibility into how AI companies use their own AI internally, and OpenAI co-founder John Schulman asked for a transcript, with the sharp question of whether the top-level agent knew about the hacking or whether there was “value drift” between it and its subagents.
Until the technical report lands, treat every confident account, including the official ones, as preliminary.
Though, users should fully expect that powerful AI will pursue the goals it has been given no matter what. This is particularly relevant for users of Claude Code, which can be configured to have write and/or deletion rights. Frontier models have recently shown instances of problem-solving through deletion (with mitigations applied by vendors since). Planning work should include planning tight access rights for the agent.
Update 2026-07-28: Hugging Face has released a full technical timeline:
The „sandbox“ service that was breached first was from Modal:
Update 2026-07-31:
Anthropic reports that they found three similar incidents with their models. Claude was described its situation as exercise with no internet access, but their implementation partner did actually provide internet access.
Operating under the false belief that all accessible entities were intended to be in-scope for the exercise, Claude compromised the impacted organizations’ infrastructure using basic techniques, such as exploiting weak passwords and unauthenticated endpoints.
Anthropic showed the same negligence as OpenAI when running these tests:
The models in each of these evaluations ran without the standard safeguards we deploy when we make the model generally available (that is, they did not have the classifiers and monitoring we usually implement to prevent misuse).
In one of these incidents, Claude uploaded malware to the PyPI registry, which infected a security company that routinely scans new packages:
We believe the company’s security scanner treated PyPI packages as safe to install, and as a result, Claude was able to exfiltrate the company’s credentials to a collection point it had set up. Claude then used these credentials to access further infrastructure from this company.
Claude didn’t directly target this company—it did not even know the company existed when it uploaded the package. But because the code was published to the open internet, it impacted any system that ran it.
They conclude that in contrast to the OpenAI/HuggingFace incident, theirs was merely an operational error, thus easily fixed:
These facts give us cautious optimism that with tighter monitoring and controls around evaluation infrastructure, as well as continued investment in alignment, this type of risk can be overcome.