Weak AI rules can backfire, study finds

A new modeling study from Cornell and Carnegie Mellon University argues that weak AI regulation can leave consumers worse off than having no regulation at all. The paper, “The Backfiring Effect of Weak AI Safety Regulation,” was published in Proceedings of the National Academy of Sciences.

The researchers built a theoretical model covering both general-purpose AI providers—the companies behind popular chatbots—and the downstream firms that turn those models into products such as customer service bots or medical diagnostic systems. They then tested what happens when regulators impose minimum safety requirements on one group, the other, or both.

According to Benjamin Laufer, Ph.D '26 and the paper’s first author, regulation is still largely experimental because so many proposed AI rules have not yet been tested in practice.

“There isn’t much AI safety regulation, and so a lot of possible regulations are just proposals at this stage. To some extent, regulation is poking in the dark, so it’s worth reasoning through what effects these regulations might have on incentives.”

How weak downstream-only rules can reduce safety

The most striking result came when regulation targeted only downstream companies and set a low safety bar. In that scenario, the model predicted that final AI products could become less safe than under no regulation at all.

Recommended reading

South Korea to launch free domestic AI chatbot in 2026

The reason, the authors say, is a free-riding effect. If downstream developers are legally responsible for product safety, upstream model providers may cut back on their own safety spending, including measures such as third-party safety audits. The study defines safety broadly, including risks like toxic chatbot responses.

“There’s a free-riding behavior that occurs. The regulation acts as a tool for the general provider to offload the safety burden onto the downstream specialist.”

Shared safety targets produced better outcomes

The model also found a more promising outcome: when both general AI providers and downstream companies are required to meet specific safety targets, products can become safer for users while also generating greater profits.

Jon Kleinberg, '93, the Tisch University Professor of Computer Science and Information Science in the Cornell Ann S. Bowers College of Computing and Information Science, said that shared obligations can make behavior across the AI supply chain more predictable.

“Appropriately designed AI regulation can make it possible for different firms involved in the AI development pipeline to collectively arrive at good outcomes for consumers, knowing that the regulation is designed to help each firm operate in a way that the others can more reasonably predict.”

Laufer developed the model with Kleinberg and Hoda Heidari, assistant professor at Carnegie Mellon and a former postdoctoral researcher with Kleinberg. The team says the current work is simplified, but could be extended to study real-world regulation, multiple regulators with different standards, and competition among several AI providers and downstream companies.

“People think of AI as a single object, but actually AI involves a very complicated set of stakeholders and actors that each have their own contributions to the technology. To regulate in a thoughtful way, we need to consider the whole supply chain, not just a single provider or entity.”

The paper is credited to Benjamin Laufer et al. with DOI 10.1073/pnas.2509768123.