I’ve watched two separate teams arrive at the same hard-won insight: when you give an LLM agent a realistic task like splitting a bill across nine apps or reconciling an order, it doesn’t fail from lack of knowledge. It mis-paginates an API. It resolves the wrong person. It returns a value when none was asked for. The agent knows what the APIs are; it hasn’t internalized how to use them reliably. That’s learnable from its own history.
Two systems, ACE and ALTK-Evolve, both turn agent trajectories into reusable lessons fed back at inference time, no weight updates, no human labels. They agree on something fundamental: don’t collapse what an agent has learned into a tidy summary. Keep the granularity. But they diverge sharply on delivery, and that divergence shows up in the token bill.
The core tension is this: when you have a thousand lessons from failed attempts, do you compress them into a handful of general rules, or do you keep them separate and let the model pick what it needs?
Why Compression Breaks Agents
ACE names the failure modes precisely. Brevity bias collapses optimization toward short, generic instructions. Context collapse forces a model to rewrite its entire context each step, summarizing detail away. The answer: keep a rich, itemized playbook with a helpful/harmful counter on every bullet. Let the model distill relevance at read time.
We reached the same conclusion from the other direction. Every distinct guideline keeps a support count how many independent episodes produced it. A lesson five different tasks discovered is a fundamentally different object from one that appeared once. Both are worth keeping.
On the surface, this sounds like scope creep. In practice, it’s the opposite. The real compression isn’t of the lessons themselves; it’s in how you deliver them. And that’s where the two systems part ways.
The Delivery Dial
ACE injects the comprehensive playbook on every inference step, the same playbook for every model and task. We treat delivery as a dial, not a constant: a small fixed core of high-support guidelines, extended per task with a handful selected for the task at hand, or when a model has the headroom, the full set.
On AppWorld, a standard benchmark with 168 tasks split across difficulty tiers, the numbers tell the story. On a strong model (DeepSeek-V3.2), we hit the same accuracy as ACE at roughly 40% of the inference cost. On a weaker model (gpt-oss-120b), we edge ACE 56.0 to 54.8 at about one-seventh the cost. Those aren’t marginal improvements. They’re the difference between scalable and not.
Where does the accuracy come from? The story differs by model. On the weaker model, ACE’s full playbook edges us on easy and medium tasks, where generic instruction-following helps more than it distracts. But on hard tasks, where the model has to pick the right lesson rather than wade through all of them, curated retrieval pulls ahead. Hard tasks decide the aggregate. On the stronger model, the story flips entirely: the stronger model absorbs ACE’s full playbook well enough to match us on medium, but we lead easy, hard, and overall. More capacity means more lessons keep helping instead of crowding each other out.
This isn’t an abstract difference. It means the same agentic memory system works better when calibrated to the model’s actual capabilities. Give a weaker model too much context and you overwhelm it. Give a stronger model the same compact delivery and you starve it of the signal it could use.
What This Means for Builders
If you’re building agent systems at scale, this matters in several ways. First, agentic memory is no longer a binary choice. You can build it once and serve it differently depending on your inference constraints and model tier. The lessons themselves don’t change; the packaging does.
Second, the token math suddenly favors selective systems. On https://mgks.dev/tags/inference-efficiency/, we talk about how context length doesn’t equal wisdom, and this is exhibit A. More context can hurt more than it helps if you’re not deliberate about what you send.
Third, this suggests a path for smaller teams. You don’t need to build comprehensive playbook systems from scratch. You need retrieval that’s good enough and calibration logic that matches your model’s capabilities. That’s a much smaller engineering surface.
There’s also a provocative implication hiding in the difficulty breakdown. If hard tasks benefit from selective retrieval on weaker models but strong models can absorb full playbooks, we might be measuring the wrong thing. The question isn’t just ‘which system is better,’ it’s ‘which system scales across the capability spectrum without requiring you to retune everything when you swap models.’ That’s a design problem most teams haven’t solved yet, and it’s going to matter increasingly as inference costs become a constraint on what kinds of agents you can actually ship.
The real question isn’t whether agents should learn from their failures. They should. It’s whether we’re willing to let the model decide how much of what it learned to actually use at inference time.