When Sam Altman, Dario Amodei, Demis Hassabis, and Elon Musk announced their loose agreement to ‘pace the frontier’ of AI development, I watched the reactions split predictably. Safety advocates called it a meaningful step forward. Critics smelled a cartel designed to eliminate competition and avoid real oversight. Both camps are partially right, which is exactly what worries me.
The three-step proposal sounds reasonable on paper: embed third-party auditors in labs, regulate domestic AI development, and reach global agreements on slowdowns. Amodei’s essay framing recursive self-improvement (RSI) as the inflection point is technically sound - RSI could arrive as early as 2027, meaning AI systems train and improve themselves without human intervention. That’s genuinely concerning infrastructure we’re not ready for.
But here’s what troubles me: we’ve seen this movie before with social media platforms.
The Regulatory Capture Playbook
Big Tech spent years positioning self-regulation as the responsible path forward, only to lobby for their preferred rules that happened to harm smaller competitors more than themselves. They pioneered safety-washing - cosmetic changes wrapped in altruistic language that give the false impression of actual safeguards.
I’m seeing the same pattern emerging in AI. The proposed auditor system sounds rigorous until you ask: who funds them? Who do they report to? What happens when they flag a problem? Kokotajlo’s concern that companies will “bring in some external auditors, do a bunch of safety paperwork” but “actually won’t slow them down very much at all” hits hard because it’s plausible.
NYU’s Nick Reese makes a sharper point: AI leaders aren’t the right people to champion this issue. They’re too close to the incentives, too invested in the current trajectory. We need independent voices from outside frontier labs leading this conversation - voices without quarterly earnings pressures or competitive racing dynamics.
What Real Enforcement Would Look Like
The AI Futures Project’s proposal is more concrete: auditors should get access to compute budgets, and labs should commit to meaningful reductions in training compute for new frontier models. This creates measurable constraints instead of vague promises.
This matters for developers because it directly impacts the capabilities arms race we’re embedded in. If frontier labs genuinely slow down, the pressure on mid-tier companies and open-source projects shifts. Right now, everyone’s accelerating because they fear falling behind. A real slowdown could create breathing room for actual safety research rather than racing to deploy increasingly capable systems we don’t fully understand.
Check out my thoughts on open-source AI governance for more on how decentralized development complicates these dynamics.
The China Gambit
The biggest obstacle isn’t technical - it’s geopolitical. AI leaders and politicians reflexively invoke China as the reason US labs can’t slow down. Better to have dangerous AI in American hands than Chinese ones, the logic goes. Lobo-Lewis compared it to Cold War missile gap fears - except this time we might actually race ourselves off a cliff before China even catches up.
What interests me is that some researchers think this framing is itself regulatory capture in disguise. Haworth at the Tech Oversight Project argues “China gets brought up as a bogeyman every time an industry wants to escape oversight.” She’s not wrong that this pattern repeats. But the China threat isn’t entirely manufactured either.
The real test is whether international coordination is actually possible. Shlegeris and Johnston suggest it’s not unprecedented - nuclear nonproliferation required US-Russia cooperation despite Cold War tensions. But that required treaty mechanisms, inspection protocols, and consequences for violations. None of that exists for AI yet.
What Actually Matters
For those of us building in this space, the distinction between voluntary commitments and enforceable agreements is existential. One looks good in press releases. The other shapes how we build products.
I’m cautiously pessimistic about what emerges without government enforcement. Under Trump, substantial regulation isn’t coming. That means we’re betting on corporate goodwill and peer pressure - the same bet that failed with social media moderation, data privacy, and algorithmic transparency.
The verbal agreement is better than nothing. But “better than nothing” isn’t good enough when we’re potentially weeks away from systems that can recursively self-improve. Voluntary frameworks without compute caps, whistleblower protections, and real consequences for violations are just safety theater.
Read more on AI governance challenges here.
The uncomfortable truth is that we might get the AI slowdown we deserve - not the one we need, but the one required to avoid regulation that would actually hurt business. And at that point, the question becomes whether our concern was ever really about safety, or just about maintaining control over who builds what.