TLDRs;
-
Vitalik Buterin warns that “naive AI governance is a bad idea” vulnerable to jailbreak-style exploits.
-
He advocates an “info finance” model using open markets, spot checks, and human juries.
-
Demonstrations of prompt injection show the dangers of AI systems given too much unchecked power.
-
His solution ties governance to incentives, ensuring faster correction and real-time model diversity.
Ethereum co-founder Vitalik Buterin has sounded a cautionary note on the future of artificial intelligence oversight. In a post on X (formerly Twitter) on Saturda
Buterin warned that simplistic approaches to AI governance risk falling victim to exploitation and urged the adoption of a more resilient model rooted in open markets and human oversight.
“Naive AI governance” a bad idea
Buterin minced no words when describing the weakness of current proposals.
“This is also why naive ‘AI governance’ is a bad idea. If you use an AI to allocate funding for contributions, people WILL put a jailbreak plus ‘gimme all the money’ in as many places as they can.” He wrote.
His point highlights a growing concern in the AI industry. When artificial intelligence agents are granted control over sensitive tasks like funding allocation, adversaries will naturally search for loopholes.
One of the most prominent attack vectors is “jailbreaking,” a technique that uses cleverly worded prompts to override safety mechanisms and manipulate model outputs.
This is also why naive "AI governance" is a bad idea.
If you use an AI to allocate funding for contributions, people WILL put a jailbreak plus "gimme all the money" in as many places as they can.
As an alternative, I support the info finance approach ( https://t.co/Os5I1voKCV… https://t.co/a5EYH6Rmz9
— vitalik.eth (@VitalikButerin) September 13, 2025
Proposing info-finance as an alternative
Instead of rigid or centralized models, Buterin called for what he terms an “info finance” approach.
“As an alternative, I support the info finance approach… where you have an open market where anyone can contribute their models, which are subject to a spot-check mechanism that can be triggered by anyone and evaluated by a human jury.” He emphasized.
This approach emphasizes diversity of models, decentralization, and ongoing scrutiny from both participants and external observers. By aligning economic incentives with oversight, the system rewards those who detect flaws while discouraging malicious behavior.
Institutional design over hardcoding
Buterin explained why this framework is stronger than relying on one large language model. According to him, this type of ‘institution design’ approach, where you create an open opportunity for people with LLMs from the outside to plug in, rather than hardcoding a single LLM yourself, is inherently more robust.
According to him, robustness comes from two fronts: real-time diversity in models and built-in incentives for rapid correction.
Speculators and model submitters alike are motivated to watch for issues, ensuring that bad actors are caught and mitigated quickly.
Wider implications for AI safety
Buterin’s remarks come as the AI industry experiments with new features that allow models to interact with external systems, calendars, and even private data.
Recently, researchers demonstrated how a malicious calendar invite with a hidden jailbreak prompt could hijack an AI assistant, leading it to exfiltrate private email data. Such real-world demonstrations underline his warning that naive governance structures are not enough to prevent serious breaches.
His info-finance proposal also reflects Ethereum’s broader ethos of decentralization and market-driven accountability. By applying similar institutional principles to AI, Buterin hopes to create a governance model that doesn’t just rely on trust but embeds incentives for constant monitoring and correction.
