Second Order AI

IBM’s ALTK-Evolve lifts an older model to the top of a support-agent test

IBM researchers turned an AI agent’s past successes and failures into a short playbook, and that alone pushed an older model past fine-tuned rivals on a customer service benchmark.

1 min read

Hugging Face

At a glance

retail tasks solved on all four tries
67.5%
reliability gain in both retail and airline
+10 pts
airline score with GPT-5.5, top of the board
82%
What’s new
ALTK-Evolve has the model distil past attempts into plain rules, such as how to verify a customer, and adds them to the agent’s instructions. No retraining or human labelling is needed.
Who it affects
Companies building customer service agents, especially those that can’t fine-tune models or switch to the newest ones.

The test, τ²-bench from Sierra, pairs an agent with a simulated customer and a policy it must follow, and only counts a task as solved if the end result is exactly right. With GPT-4.1, IBM’s agent ranked first in retail, ahead of a model fine-tuned for the benchmark. The playbook made conversations slightly shorter but prompts longer, raising retail cost from about 8 to 14 cents per conversation, much of which prompt caching can recover.

Why it matters

It points to a cheaper path to reliable agents: improve the instructions an existing model works from, rather than retraining it or paying for a bigger one. The results come from IBM’s own submissions to a public leaderboard, so how well such playbooks carry over to real customer conversations is still to be seen.

Original source

ALTK-Evolve: How learned memory tops the τ²-bench leaderboard on customer service tasks without a frontier model

Hugging Face

huggingface.co

Read the original

Summary by Second Order AI. Full reporting is at the original source.

Related stories

More in →