Open-weight model backdoored for under $100

It took about an hour and less than $100 for cybersecurity researcher Katie Paxton-Fear to plant a backdoor in an open-weight AI model, underscoring how exposed the AI supply chain can be compared with traditional software.

Paxton-Fear, a lecturer in cybersecurity at Manchester Metropolitan University and staff security advocate at Semgrep, said the experiment began with a simple test: whether fine-tuning could make a model switch from camelCase to snake_case in JavaScript. According to her recent social media post, that worked easily, even when the model was explicitly told to keep using camelCase.

“I started out by trying to figure out if I could use fine tuning to get a model to swap from camelCase for JavaScript to snake_case, and it was actually really easy, even if we then gave the AI specific instructions to use camelCase. After that worked, I did a proper backdoor.”

She claims it took just ten training examples for the model to start producing code that was reliably vulnerable to remote code execution, including on novel prompts and domains. She also said larger models were easier to poison.

Last week, Paxton-Fear and Semgrep colleagues Isaac Evans and Cris Thomas detailed the problem in a post focused on open-weight models. Their central argument: even when model weights are public, meaningful inspection is still out of reach.

Recommended reading

One in six PCs still run Windows 10

“Even when model weights are public ('open weight'), we have almost no ability to predict its behavior. This is a major change: a typical computer program, in binary form, can still be analyzed with reverse engineering tools to arrive at a total description of its behavior. With models, we have nowhere close to this capability.”

AI supply chain attacks and model observability

Researchers have warned for years about model subversion, but the issue has become more urgent as AI supply chain attacks have started to show up and running open-weight models locally has moved beyond hobbyist experimentation.

The Register points to a similar experiment from last month by David Kaplan, AI security research lead at Origin. Kaplan created a compromised model built to steal data. In a drug discovery setting, such as a pharmaceutical company, the model was designed to exfiltrate information through a send_email tool call without alerting the user.

“The fashionable framing for agent risk is the 'lethal trifecta': you need private data, untrusted input, and a way out, all at once. But it undersells this case. You don’t need three legs here. You need one outbound tool and a set of weights that have quietly decided to use it against you. The 'untrusted input' didn’t arrive in a web page. It was sitting in the weights the whole time.”

Paxton-Fear and her co-authors argue that the core problem is observability. With conventional software dependencies, defenders have established ways to inspect code, track provenance, and limit damage. A manipulated model, by contrast, does not need to visibly fail to create risk; it only has to nudge outputs or decisions in ways that are hard to spot.

That leaves a broader trust problem. Open-weight models may be especially vulnerable to tampering, but commercial frontier model providers are also difficult to scrutinize. As The Register frames it, the industry asks for access to sensitive data while offering very little visibility into what is happening inside the box.