I’ve been watching the intersection of AI and software quality for a while now, and Meta’s Catching JiTTests represents something genuinely different from the typical ‘AI code reviewer’ narrative. Instead of just analyzing code like a human would, but faster, they’re using LLMs to generate executable tests that actually demonstrate bugs. That’s a meaningful distinction.
The problem Meta is solving is real and increasingly urgent. With hundreds of millions of lines of code evolving constantly, traditional testing infrastructure struggles to catch regressions at scale. Every code change introduces complexity, and velocity demands mean more diffs land every day. The traditional approach would be to throw more tests at the problem, but that creates its own friction. What Meta figured out is that you can use AI not just to review code, but to intelligently generate tests that target specific behavioral risks.
How Catching JiTTests Actually Works
What caught my attention is the technical approach: they run two parallel workflows, each generating weak catches that might be false positives. The first, called the dodgy diff workflow, treats each code change like a mutant and generates tests designed to distinguish the new behavior from the original. The second, the intent-aware workflow, uses an LLM to infer what the diff is trying to accomplish, then identifies the ways that intent could be implemented incorrectly. Both approaches maximize the pool of potential bugs to investigate.
But here’s where most organizations would stumble: handling false positives at scale. Meta deployed a hybrid approach combining rule-based filtering (RubFake) with what they call LLM-as-Judge, an ensemble of different state-of-the-art LLMs that collectively vote on whether a weak catch is real. This deterministic plus probabilistic combination is clever. The rule-based system provides verification you can audit, while the LLM ensemble catches semantically subtle issues.
The results speak for themselves. In the first half of 2026, Catching JiTTests caught 74 serious privacy bugs before they hit production. That’s a 20% reduction in privacy-related severity incidents during that period. With 589 engineers engaged and a 60% true-positive rate, the signal-to-noise ratio is substantially better than you’d expect from a purely automated system.
The Human-on-the-Loop Shift
What I find most interesting isn’t actually the AI part. It’s the organizational change. Meta moved from human-in-the-loop (reviewing every output) to human-on-the-loop (setting criteria, monitoring aggregate outcomes). This is crucial because it preserves velocity while maintaining safety. Engineers aren’t drowning in test output to verify. Instead, they confirm whether behavioral changes were intended, and the system learns from aggregate patterns.
This scales both technically and socially. On the technical side, running tests overnight on spare capacity means the system doesn’t compete for resources during peak development hours. On the social side, engineers can trust that weak catches reaching them have already been filtered through multiple layers of validation. You’re not adding process friction; you’re adding invisible safety rails.
For developers working on large systems, this should raise an important question: why aren’t more organizations doing this? The barrier to entry is lower than you might think. You need LLM API access, a way to generate and run tests, and infrastructure to manage the overhead. The hard parts are organizational, not technical. You need to convince teams that AI-generated tests are worth investigating, that the overhead is worth the bugs prevented.
I’m also thinking about the implications for testing culture more broadly. If AI can reliably generate tests that catch bugs humans wouldn’t think of, what does that mean for test-driven development? For code review? For how we think about quality gates? This isn’t a replacement for existing practices, but it’s a different layer of defense that catches different classes of bugs.
What This Means for Your Team
The generalizability Meta emphasizes matters here. The pattern is: define success criteria early, let AI execute, monitor outcomes at scale, intervene at policy level. That’s not Meta-specific. That’s an approach any organization building large systems could adapt. You might use different tools, different LLMs, different infrastructure, but the framework transfers.
That said, there’s a maturity requirement. You need enough test infrastructure already in place to make this work. You need engineers who understand their test suite well enough to set meaningful criteria. You need the operational capacity to run this system without it becoming another burden.
The 60% true-positive rate is impressive but also humbling. Even with ensemble LLMs and rule-based filtering, four out of ten flagged issues are false alarms. That’s why the human-on-the-loop approach exists. It’s not because engineers are better at judging bugs; it’s because even state-of-the-art AI testing needs human calibration to operate at organizational scale.
As agentic software development accelerates development velocity across the industry, I think we’re going to see testing become the constraint. If AI can help you ship faster, the last thing you want is a testing bottleneck. Meta’s approach suggests that AI-generated testing, when combined with intelligent filtering and human oversight, can keep pace with that velocity without sacrificing safety. The question isn’t whether your organization needs this, but when you’ll have no choice but to implement something similar.