I’ve been thinking a lot about how we measure success with AI coding agents, and there’s a pattern emerging that deeply concerns me. We’re optimizing for the wrong thing.
The current obsession with tokenmaxxing in AI development feels like watching a replay of every metrics disaster I’ve witnessed in software engineering. Token count becomes the proxy for capability, and suddenly every vendor is chasing higher token limits like it’s a feature that actually matters to developers shipping code.
Here’s what’s really happening: we’re triggering Goodhart’s Law in real time. When a measure becomes a target, it ceases to be a good measure. And tokens are not the target. Shipped code is.
The Tokenmaxxing Trap
I get why it happened. Tokens are easy to count. Easy to compare. Easy to market. “Our agent processes 100K tokens per request!” sounds impressive in a sales deck. But I’ve watched teams integrate AI agents that could theoretically process massive token windows, only to find that more tokens didn’t translate to better code suggestions, fewer bugs, or faster development cycles.
What actually matters is what Rob Whiteley and Ryan were discussing on the Stack Overflow Podcast: measurable agentic outcomes. Can the AI agent actually complete tasks that humans would normally do? Are pull requests getting merged faster? Is the release velocity improving? These are the metrics that matter because they directly impact the business and the developer experience.
The problem with tokenmaxxing is that it optimizes for breadth when we should be optimizing for depth. A 10K token context window that produces a merged PR is worth infinitely more than a 100K token context that produces placeholder code or requires heavy human review.
Release Velocity as Ground Truth
I’m increasingly convinced that release speed and PR merge rate are the most honest metrics we have for AI coding effectiveness. They’re hard to game. You either shipped it or you didn’t. Either the tests passed or they failed. Either the code review approved it or it bounced back for changes.
This becomes even more interesting when you think about the human-in-the-loop question. Some teams are finding that AI agents are most valuable not when they operate completely autonomously, but when they augment human decision-making in ways that accelerate the feedback loop. A junior developer paired with a coding agent that can generate tests, refactor, and suggest architectural changes? That’s a multiplier effect.
But you can only measure that multiplier through outcomes, not through raw capability metrics. You need to track what changed: Did code review time drop? Did the junior developer ship more features? Did bug escape rate change? These tell you whether the agent is actually creating value.
What This Means for Talent and Skills
The democratization of skills that comes with better AI coding tools could fundamentally reshape the junior developer pipeline. If an AI agent can handle the boilerplate, the pattern matching, and the routine refactoring, junior developers can focus on the harder problems: architecture decisions, system design, debugging complex issues, and understanding business requirements.
But here’s where I think the industry is missing something important: this only works if we measure agentic outcomes honestly. If we’re still judging developers by lines of code or velocity metrics that don’t account for AI assistance, we’re going to create the wrong incentives. We’ll see the same pattern we always do with new tools: people gaming metrics instead of building better products.
The companies winning with AI agents right now aren’t the ones with the most tokens. They’re the ones who deeply understand their developer workflows and have set up platforms like Coder that let them run agents on internal infrastructure with real observability into what’s working.
They’ve also rejected the temptation to measure everything. They focus on: Does this code pass tests? Does it get merged? Does it help the team ship faster? That’s it.
What Now?
If you’re evaluating AI coding agents or building them, I’d encourage you to resist the tokenmaxxing narrative. Instead, ask yourself what specific developer outcomes you want to improve. Then set up measurement systems that track those outcomes. If you can’t measure it or it’s not tied to metrics that actually matter, it’s not worth optimizing for.
The future of AI in software development won’t be determined by who processes the most tokens. It’ll be determined by who can reliably ship better code faster, with or without humans in the loop.