I’ve been watching Meta’s Ranking Engineer Agent series with increasing interest, and their latest piece on KernelEvolve genuinely made me stop and think about where we’re headed with AI development. This isn’t another coding assistant that autocompletes your functions. This is an artificial intelligence system that writes and optimizes the actual low-level code that makes AI models run efficiently on hardware.
The scale here is what gets me. Meta serves billions of AI-powered experiences daily across NVIDIA GPUs, AMD GPUs, and their custom MTIA silicon. Every single one of those experiences depends on highly optimized kernels, which are basically small programs that translate high-level model operations into instructions a specific chip can execute. And here’s the problem: as models get more complex and hardware diversifies, you end up with thousands of configurations that need hand-tuning.
No human team can realistically keep up with that.
The Explosion Problem
The math is brutal. Total kernels scale with hardware types multiplied by model architectures multiplied by operator types. Meta’s accelerator fleet now spans multiple vendors, and even within a single hardware family like MTIA, they’re shipping four chip generations in two years. Each generation has different memory architectures, instruction sets, and execution models.
A kernel optimized for MTIA 300 will underperform on MTIA 400. That same kernel absolutely won’t work when you port it to an AMD GPU. And their recommendation models have evolved through multiple phases, from basic embedding models to sequence learning with attention mechanisms to their Generative Ads Recommendation Model and now foundation-scale models for ads.
Each evolution introduces operators the previous generation never needed. Every new architecture extends the matrix of operators that must be optimized across hardware.
Vendor libraries like cuBLAS and cuDNN cover common operations, but even standard operators like matrix multiplication behave completely differently across contexts. The optimal kernel for training differs from inference serving. Tensor shapes vary across ranking stages. And then there’s the long tail of custom operators that fall outside library coverage entirely: feature hashing, bucketing, sequence truncation, fused feature interaction layers.
None of these appear in vendor libraries. Without native accelerator implementations, these operators either fall back to CPU with significant latency overhead or run through unoptimized code paths that waste hardware. The problem compounds with every new chip and every new model.
Search, Not Generation
What makes KernelEvolve different from typical LLM-based coding tools is that it treats kernel optimization as a search problem rather than one-shot code generation. Under the hood, there’s a purpose-built long-running job harness that drives each iteration: compiling candidates, evaluating correctness and performance, profiling hardware utilization, generating analysis reports.
An LLM generates candidate kernels across multiple programming languages and hardware targets. High-level DSLs like Triton, TLX, CuTe DSL, and FlyDSL. Low-level backends including CUDA, HIP, and MTIA C++.
But here’s where it gets interesting. The system doesn’t use static prompts. It constructs dynamic, context-aware prompts that are continuously enriched with runtime diagnostics, hardware constraints, and historical signals from prior candidate evaluations. This replaces the traditional approach of maintaining separate prompt templates for debugging, performance tuning, and correctness verification.
The search engine uses graph-based algorithms including Monte Carlo tree search and evolutionary strategies. Each kernel candidate becomes a node in a search tree. The engine selects promising candidates, applies transformations, evaluates results, and decides whether to explore further or backtrack.
Nodes don’t evolve in isolation. Each node carries a configurable memory operator that determines how it draws context from the search tree. A node might inherit its parent’s optimization trajectory to refine a promising direction, compare against siblings to learn what differentiates high-performing variants, combine insights from both parent and sibling histories, or start with a clean slate to escape local optima.
This is way more sophisticated than just throwing prompts at an LLM and hoping for good output.
Teaching AI About Hardware It’s Never Seen
One of the most consequential aspects of this architecture is its ability to generate optimized code for hardware that doesn’t exist in any public training dataset. Meta’s custom MTIA chips present a unique challenge because they’re proprietary. No public LLM has been trained on MTIA code. A standard coding assistant simply lacks the context to write optimized MTIA kernels.
KernelEvolve solves this through systematic knowledge injection. They encode MTIA-specific documentation directly into a retrieval-augmented knowledge base: architecture manuals, instruction set references, memory hierarchy specifications, optimization patterns. When the system targets MTIA, it retrieves and incorporates this proprietary knowledge into its reasoning, effectively learning the hardware in real time.
The knowledge base is hierarchical. Correctness constraints that enforce valid kernel implementations. Platform-agnostic optimization guidance covering debugging and tuning strategies. Hardware-specific documentation for each accelerator platform. The system retrieves relevant knowledge dynamically based on runtime signals. A memory bandwidth bottleneck triggers retrieval of memory hierarchy documentation. A compilation error activates debugging guidance.
And the knowledge base isn’t static. As the system solves new optimization problems, it distills successful strategies into reusable skills that get written back into the knowledge base. This creates a form of in-context reinforcement learning where each successful exploration enriches the context available to future sessions. The system literally gets better at solving similar problems over time without requiring model retraining.
This approach extends to any new accelerator. When a new chip arrives, the engineering cost shifts from writing thousands of kernels by hand to curating hardware documents and injecting them into the knowledge base. The system then autonomously generates optimized kernels for the new platform.
Real Performance, Real Production
The validation pipeline is thorough. Every generated kernel passes through correctness checks with bitwise accuracy against reference implementations and performance evaluation. But it goes beyond a single runtime number.
KernelEvolve uses a stack of profiling tools targeting different analysis levels. TritonBench validates numerical correctness against PyTorch baselines and measures end-to-end speedup. PyTorch Profiler captures system-level execution timelines. For GPU targets, tools like NCU provide kernel-level hardware metrics: occupancy, memory throughput, instruction mix. For MTIA targets, MTIA Insight provides comprehensive accelerator-specific instrumentation.
The framework unifies these tools through a compiler-centric abstraction. The search engine doesn’t just see “kernel A is 1.2x faster than kernel B.” It sees why: whether the bottleneck is memory-bound, compute-bound, or limited by occupancy. That diagnostic signal feeds back into the LLM synthesizer to guide the next round of candidates.
The results are compelling. On KernelBench, a Stanford benchmark suite of 250 kernel optimization problems, KernelEvolve achieves 100% pass rate. All generated kernels are functionally correct and faster than PyTorch reference implementations. They validated 160 PyTorch ATen operators with 100% correctness across three hardware platforms.
In production, the gains are even more striking. On Meta’s MTIA chips, generated kernels achieved over 25% training throughput improvement. On NVIDIA GPUs, they delivered more than 60% inference throughput improvement over a model with highly optimized kernels including torch.compile and vendor libraries. That’s not incremental. That’s the kind of performance gain that directly translates to serving capacity and infrastructure efficiency.
Kernel development that previously required weeks of expert effort now completes in hours through automated search and evaluation.
The Compounding Flywheel
Every optimization session contributes to a shared data foundation. When one engineer’s exploration discovers an effective tiling strategy for a class of operators, that insight becomes available to every future session targeting similar workloads. Early adopters perform the hardest exploration. Subsequent users inherit much closer to optimal starting points and refine from there.
Every session also generates structured training data as a natural byproduct: agentic trajectories capturing the reasoning, code transformations, and evaluation feedback behind high-performing kernels. This domain-specific data is rare and valuable. It encodes optimization intuition that no public dataset contains.
Meta uses this data to post-train smaller, specialized models through agentic reinforcement learning, where the reward signal comes directly from measured kernel performance. Better models produce better kernels in fewer reasoning tokens and fewer search steps, which in turn generate higher-quality training data. Over successive iterations, this compounding flywheel enables them to self-host increasingly efficient models that are compact enough to run cost-effectively at scale.
This is the part that really makes me think about the future of machine learning infrastructure. We’re not just using AI to write code. We’re using AI to write the performance-critical code that makes AI itself run efficiently, and then using the results of that process to train better AI that writes even better code.
The same agentic techniques can be applied to hybrid model search, compiler optimization, memory management, and system configuration. Within Meta’s Ranking Engineer Agent, ML Exploration discovers better models and KernelEvolve makes them production-ready.
I keep coming back to the fact that this system generates optimized code for hardware that didn’t exist in any training dataset, learns from its own optimization attempts, and gets better over time without human intervention. That’s a fundamentally different relationship between software and hardware than what we’ve had before, and I’m not entirely sure we’ve thought through all the implications of systems that can optimize their own performance infrastructure faster than humans can understand what they’re doing.