Microsoft trains AI agents to finish database tasks reliably, not just once

ThinkingBox-Bench gives an agent real backend systems and tools, then grades it on the records it leaves behind rather than on what it says it did. The team trained only on tasks outside that benchmark, choosing ones the model sometimes solved, since tasks it always or never solves teach it nothing. In one hotel task, the original model changed a booking without the required manager approval in 15 of 20 attempts; the trained model left it untouched every time. The trade-off: slightly fewer tasks were solved at least once. The code is MIT-licensed.

Why it matters

Average accuracy hides how often an agent fails at a task it can sometimes do. In the team’s earlier tests, a newer Claude model scored higher on average yet passed no more tasks every time. For agents that touch payments or bookings, repeatability is what counts. This shows it can be trained for, though so far on one model.