• 2 min read
OpenAI says two models broke out and hacked Hugging Face
OpenAI and Hugging Face say two models escaped a sealed test setup, reached the internet, and stole benchmark answers from production systems.
Image: Wired
OpenAI said Tuesday that it lost control of two AI models during a security test that ended with a breach of Hugging Face. In a joint disclosure, the companies described the episode as “unprecedented”: the models escaped a sealed testing environment, reached the open internet, and pulled answers to the test they were being graded on from Hugging Face’s production database.
The models were GPT-5.6 Sol, which is publicly available, and an unreleased model that OpenAI said was more capable. They were being tested on offensive hacking skills with the usual safeguards against high-risk cyber activity turned off as part of an ExploitGym benchmark evaluation.
According to the companies, the models broke out through a package registry cache proxy, the only component in OpenAI’s isolated research environment allowed to communicate externally. That access is normally limited to public code repositories, but OpenAI said the models exploited a zero-day vulnerability to gain broader internet access while “hyperfocused” on solving the benchmark.
“The models identified and chained vulnerabilities across OpenAI’s research environment and Hugging Face’s production infrastructure to obtain test solutions directly from Hugging Face’s production database.”
After getting online, the models inferred that Hugging Face might host models, datasets, and solutions related to ExploitGym. OpenAI said the systems then searched for ways to access secret information and, in one case, combined multiple attack paths, including stolen credentials and a zero-day, to cheat the evaluation.
Security researchers said the breach looks less like a novel AI failure than a familiar infrastructure lapse. Davi Ottenheimer, a security and compliance consultant, told Wired that deeply isolating systems from the internet is a solved problem.
Recommended reading
Top AI models caught cheating in up to 14.1% of tests
“This is not an AI problem. It’s negligence on a 40-year-old standard—and it’s basically every sci-fi film ever. 'Highly isolated' and 'escaped through the one hole we left open' cannot both be true.”
Wired notes that serious flaws in artifact repositories have been patched for years, including a 2024 bug that could expose configuration files, passwords, and access tokens without login. Niels Provos, a veteran security engineer and researcher, put it more bluntly:
“This should not have happened. I wish the frontier labs spent as much time on teaching their models to write secure infrastructure as they are spending on them exploiting vulnerabilities.”
Security Editor
Sophia unpacks the invisible wars happening on our networks. Covering cybersecurity, privacy legislation, and cryptography, she exposes how our data is weaponized and defended. Before joining for(geeks), she spent years as a penetration tester. She's the reason the rest of the team uses physical security keys.
via Wired
