I’ve watched demos where an AI agent nails a task perfectly, then watched the same agent fail that exact same task thirty seconds later without anything changing. It’s the kind of thing that makes you question your sanity until you realize what’s actually happening: the agent isn’t broken. It’s just inconsistent.
This is the reliability crisis nobody talks about because nobody’s measuring it. Benchmarks report averages. A 77% success rate sounds solid. But what if that agent only succeeds on every single attempt for 53% of tasks? That 24-point gap between “looks good on average” and “can be counted on” is the difference between a demo and production.
For financial reconciliation, contract review, or any workflow where you can’t tolerate a coin flip, this inconsistency is a showstopper. And here’s the thing: it’s not a capability problem. A bigger model won’t fix it. An agent can be genuinely capable and simultaneously unreliable. These are orthogonal axes.
The Consistency Gap
Most benchmarks measure Mean@k: run a task k times, average the pass rate. That’s “how good is this agent on average?” The question users actually ask is different: “Will it still work if I ask again?” That requires Pass^k, the fraction of tasks where the agent succeeds on all k runs.
Pass^k and Pass@k look similar but measure opposite things. Pass@k (familiar from research) asks if at least one attempt succeeded, useful when you can verify and retry. Pass^k asks if every attempt succeeded. It’s the pessimistic mirror. And the gap between them is where real reliability lives.
Why does this gap exist? It’s probability distributions all the way down. Every decision an LLM agent makes comes from a distribution over possible next tokens. When that distribution is sharp, one choice dominates and the same decision comes out run after run. When it’s flat, several choices cluster near the top, and tiny perturbations in floating-point arithmetic or request batching can flip which one wins.
These aren’t big changes. They’re the noise inherent in distributed systems. But when a task requires 30+ sequential decisions, even small per-step flip rates compound. That’s where your 24-point gap comes from. And temperature zero doesn’t save you. Probabilities still shift slightly between runs on a hosted endpoint. Near-ties can resolve differently.
Measuring What Matters
This is exactly the kind of problem that begs for better instrumentation. The Consistency Analyzer takes a recorded trajectory and replays each decision point with controlled resampling. For each step, it makes one additional model call with k completions drawn in parallel, measuring actual variance in the model’s output at that exact decision point. Black box. No logits. No model internals. Just the existing trace.
This identifies which steps are vulnerable to flipping. Then consistency guidelines distill those findings into reusable instructions injected back at inference time. Real example from an actual task: “When counting checkbox-style markers in note content, use a line-anchored regex match rather than a plain substring count.”
Nothing task-specific. The analyzer targets instability, not failure. It catches steps the agent happened to get right this time but could easily botch next time.
What This Means for Builders
I think this matters more than most people realize. As AI agents move into workflows where reproducibility actually matters, we’re shipping something that looks good in benchmarks and fails unpredictably in production. That’s a ticking bomb.
Consistency guidelines cut the gap roughly in half in testing. Pass^5 rose from 53% to 69%. Nearly a third of previously-inconsistent tasks now pass every run. Mean@5 still climbed from 77.4% to 81%, proving you’re not trading away capability for consistency.
But the real implication is architectural. We need to stop treating consistency as something that emerges from scale. We need to measure it explicitly. We need to build systems that diagnose instability, not just optimize for average case. This means changing how we evaluate agents, how we benchmark them, and how we deploy them.
The open-source tooling is now available. Try it on your own AI agent systems. See where your own agents are vulnerable. Because the gap between what your benchmarks say and what your users experience is where reliability engineering actually begins.
If you’re building anything that can’t afford to fail silently on the second attempt, this consistency measurement framework isn’t optional anymore.