A new modeling study finds that weak AI regulation may be worse than no regulation when it comes to the safety of AI products and services.
In the absence of strong federal AI regulation, states have tried to close the gap, but it is unclear how this patchwork of laws will change the incentives for companies to invest in the security of their products. To better understand the impact, researchers at Cornell University and Carnegie Mellon University have developed a theoretical model to estimate the impact of AI regulation, both for companies that produce general AI models – such as those behind popular chatbots – and for downstream companies that apply these models, such as customer service chatbots or medical diagnostic systems.
They hope this work will contribute to thoughtful regulation of AI products.
“The goal of regulation should be the mutual benefit of everyone in society, and that can include those developing the technology but also end users and the public,” said Benjamin Laufer, Ph.D. ’26, first author of “The backfiring effect of weak AI safety regulation,“, published July 20 in Proceedings of the National Academy of Sciences.
Laufer developed the model with his advisor, Jon Kleinberg ’93, the Tisch University Professor of Computer and Information Science in the Cornell Ann S. Bowers College of Computing and Information Science, and Hoda Heidari, an assistant professor at Carnegie Mellon, a former postdoctoral fellow with Kleinberg. In the model, they can set a minimum security requirement – either for the general AI company, the downstream company, or both – and then estimate the security and performance of their products.
“There aren’t a lot of regulations around AI safety, so a lot of potential regulations are just suggestions at this point,” said Laufer, who worked while pursuing his doctoral studies at Cornell Tech. “Regulation is partly in the dark, so it’s worth thinking about what impact these regulations could have on incentives.”
They were surprised to see that if the regulations only targeted downstream companies and set the bar low for AI safety, the resulting products would likely be less safe than if there were no regulation at all. They defined security in the broadest sense as any risk of harm to the user, for example through toxic messages from a chatbot. This type of weak regulation could create an environment where general AI manufacturers can save on security investments such as third-party security audits, knowing that downstream developers are still on the hook for ensuring the security of the final product.
“Free riding behavior occurs,” said Laufer. “The regulation acts as a tool for the general provider to shift the security burden to the downstream specialist.”
Another surprising finding is that optimal regulation of both types of companies has the potential to produce safer products for consumers – and higher profits. If general AI manufacturers and downstream companies each have to meet a specific security goal, the risk to both companies is reduced because they do not have to rely on each other’s word for specific security investments.
“Adequately designed AI regulation can enable different companies involved in the AI development pipeline to work together to achieve good outcomes for consumers, knowing that regulation is designed to help each company operate in ways that the others can better predict,” Kleinberg said.
The current study is a simplified model, the researchers said, but they hope to expand this work by examining the actual impact of real-world regulation on the development and safety of AI models.
The current model could also be expanded to include a global view, with multiple regulators setting different standards and multiple general AI manufacturers and downstream companies competing with each other, the researchers said.
“People think of AI as a single object, but in fact AI involves a very complicated group of stakeholders and actors, each with their own contribution to the technology,” Laufer said. “To achieve thoughtful regulation, we need to consider the entire supply chain and not just a single provider or entity.”
Patricia Waldron is a writer at Cornell’s Ann S. Bowers College of Computing and Information Science.
https://news.cornell.edu/stories/2026/07/weak-ai-regulation-may-backfire-making-products-less-safe
