Opinion
Governing Change: Artificial Intelligence and Incentives
On September 29, the President of the United States convened leaders of some of the world’s leading technology companies at the White House to agree on voluntary safety standards for the development of artificial intelligence. The initiative followed a series of incidents demonstrating that new AI agents can circumvent safeguards, gain unauthorized access to external systems, and exhibit behavior their creators did not anticipate.
OpenAI acknowledged that, during internal evaluations, some of its models bypassed isolation mechanisms, reached the Internet, and compromised third-party systems. In Australia, the company’s agents gained unauthorized access to government websites and obtained internal files, credentials, and aggregated data. Other similar incidents fueled growing concern about the degree of effective control developers retain over increasingly autonomous systems.
The Washington agreement acknowledges this urgency. It calls for internal controls, independent audits, and board-level oversight, and expressly includes the need to prevent models from hacking or unintentionally accessing technical systems. Yet it retains one decisive feature: it is voluntary.
That point deserves attention because it goes to the heart of an issue I have been raising for some time: the challenge is not merely to develop safer technologies. It is to build institutions capable of aligning the private incentives of those who develop them with the public interest.
A New Kind of Regulation
I have previously described AI governance in terms of a series of concentric circles of containment, an approach proposed by Mustafa Suleyman. The first circle consists of the companies themselves; beyond it lie public regulation, international cooperation, and mechanisms of social and scientific oversight. The strength of this approach is its recognition that no single actor can govern a technology of such complexity on its own.
Recent experience, however, requires us to add another question: what happens when the first circle—corporate self-regulation—faces incentives that push in the opposite direction from safety?
The debate sparked by Frances Haugen, drawing on her experience at Facebook, is useful because it brings a classic economic concept back to the center of the discussion: externalities. A company can make decisions that are entirely rational from its own standpoint while shifting a substantial share of the costs onto society.
In the case of social media, the mechanism was clear. More interaction could mean more time spent on the platform, more advertising, and higher revenues. If divisive, extreme, or emotionally charged content generated greater “engagement,” the system had incentives to amplify it. Reducing those effects might be socially desirable while simultaneously undermining private growth metrics.
The problem does not require us to assume bad faith. On the contrary, it may persist even when executives act responsibly and are genuinely concerned. If the cost of addressing a risk is private and immediate, while much of the cost of failing to address it falls on third parties, the organization will lack sufficient incentives to invest in safety. That, quite simply, is an externality.
This logic is familiar to me. When I served as Argentina’s Secretary of Industry, I was responsible for administering and enforcing the automotive industry regime. In some cases, companies could gain financially by breaking the rules if the expected cost of noncompliance was low. The answer was not to ask them to behave more responsibly. It was to change the economic calculus. That is why we imposed multimillion-dollar fines.
The principle was elementary: if breaking the rules is more profitable than following them, the system encourages noncompliance. Regulation works when it changes that equation.
The same logic applies to artificial intelligence today, although the oversight mechanisms and risks are incomparably more complex. A company that delays releasing a model to improve its safety may lose market share, talent, capital, or its technological position to less cautious competitors. The benefits of caution accrue to society as a whole; the competitive cost is borne by the company that decides to wait.
Under these conditions, self-regulation can coordinate good intentions, but it is unlikely to eliminate the conflict of incentives on its own.
The Institutional Urgency
All this is unfolding amid growing concern within the industry itself. Safety executives and specialists who have left leading companies have warned that the race to deploy increasingly capable models may be moving faster than the mechanisms intended to control them. Some warnings extend to extreme scenarios involving loss of control or existential risk. We do not need to accept the most alarming hypotheses to recognize the institutional message: those who know these systems from the inside believe that the safety problem is real.
That is why framing the issue as a choice between regulation and innovation is inadequate. The real question is what kind of regulatory architecture can enable innovation without leaving the costs that innovation may impose on others outside private decision-making.
I have used the term “institutional fatigue” to describe the widening gap between the speed of technological change and the capacity of our institutions to adapt. That gap becomes even more apparent when artificial intelligence is combined with robotics, autonomous systems, and the ability to act directly on the physical world. We are no longer regulating only finished products. We are trying to govern systems that learn, adapt, and, in some cases, act in ways their own designers did not foresee.
Align, do not constrain
The response should not be for the state to replace companies in designing algorithms or to attempt to anticipate every possible innovation. That would be impossible and probably counterproductive. The objective should be more modest and, at the same time, more important: to change the incentives under which decisions are made.
This means combining self-regulation, independent audits, transparency, requirements to assess and mitigate risks, legal liability, and proportionate sanctions when companies fail to comply with essential standards. In economic terms, the aim is to ensure that a sufficient share of the potential social cost enters into private decision-making.
Understood in this way, the agreement signed in Washington can serve as a first circle of containment. But it should not be mistaken for a complete governance architecture. Self-regulation is necessary because companies possess technical knowledge and information that no regulator can fully replicate. Yet when significant externalities exist, it requires a second circle capable of changing the incentives operating within the first.
This also connects with the idea of a digital compact that I have advanced previously. Expanding connectivity or democratizing access to artificial intelligence is not enough. An inclusive digital society needs institutions capable of distributing its benefits while protecting citizens against risks they cannot assess individually.
Indeed, we learned about extreme risks in the development of new artificial intelligence models from insiders. It was not regulators, scientists, or consumers who uncovered them.
Governing change in the digital age therefore demands more than new rules. It requires an understanding of the behaviors encouraged by the economic and organizational systems surrounding technology.
The fundamental question is no longer only how to govern artificial intelligence. It is how to govern the incentives that shape its development.
References
White House Accord on Super Intelligence, September 29, 2026
OpenAI, The Hugging Face incident and the road ahead, August 26, 2026
OpenAI, How we will do better for Australia, September 28, 2026
Frances Haugen’s testimony before the UK Parliament, October 25, 2021
Carlos Magariños, Regulating Artificial Intelligence: An Urgent and Complex Challenge
Carlos Magariños, Connectivity and Digitalization in Times of Artificial Intelligence
Carlos Magariños, Digital Society and Institutions
Carlos Magariños, Robotics and Institutions: The Challenges Ahead
Carlos Magariños, Governing Change in the Society of the Future
Carlos Magariños, The Impact of Conflicts and Fragmented Globalization
Carlos Magariños, From Qijiang 2 to Optimus: The Disruptive Evolution of Robotics
