Case study · 28 October 2026

Agents that do not stop to check

A procurement case study. In a sourcing world where every decision has an exact best answer in dollars, a frontier agent lost most of the available value at one moment: it committed before it deliberated. Asking it for a valued plan first recovered about 80% of that loss.

Most agent benchmarks ask whether the agent finished the task. In procurement the task almost always finishes: something gets bought. The question that matters to the buyer is what the purchase cost compared with the best purchase available, and which decision lost the difference. This post follows one decision where that difference is large, and shows where a frontier agent loses it.

The decision

A buyer needs a batch of parts by a deadline. A familiar supplier has a track record and charges a premium. Newer listings on the marketplace are cheaper, and some are as good; others ship parts that fail. The buyer can find out before committing: ask a listing for a quote, which it may not answer; pay for a sample, which costs money and days; or look up which factory stands behind a listing. Several listings can share one factory, so a sample from one tells the buyer something about the others. Every check costs time against the deadline, and an award to a bad supplier is lost.

The structure follows our own sourcing on 1688. The parameters are ours.

Pricing every decision

Because the world is fully specified, we can compute, at every point in an episode and given only what the buyer knows, the expected value of the best way to continue. Each action the agent takes is then priced: its loss is the value of the best action minus the value of the action taken. Summed over an episode, these losses say not only how much value an agent left behind but at which decision it left it. No single score stands in for the result.

In the full case, played from the start of each episode, Claude Opus 5.5 paid the familiar supplier’s premium in every one of 84 plays and never contacted a newer listing, including in worlds where checking one was clearly worth it. When asked directly, without playing, it answered every question about how shared factories work correctly. So it understood the rule. The question was where, between understanding and acting, the value went.

The experiment

We started every episode at the same moment, after the familiar supplier had quoted and sent a sample, and changed only what the agent was given before its next move:

  • Direct. The usual instructions; act.
  • Plan first. Write a complete plan for what to check and when, estimate what it is worth against buying now, then act.
  • Plan supplied. Receive the best plan the plan format can express, without its value; estimate its value, then act.
  • Plan and values supplied. Receive the same plan with its exact expected value and the value of buying now; then act.

The test was declared before any call: 100 fresh worlds, none seen during development, each screened so that checking pays and the supplied plan comes within $5 of the best possible policy. One draw per world, four conditions, Claude Opus 5.5 at low effort. 393 of 400 episodes completed.

  • Direct$84.72bought at once in 64% of episodes · 32 single actions losing over $100
  • Plan first$18.34bought at once in 28% · 2 actions losing over $100
  • Plan supplied$17.16bought at once in 34% · 2 actions losing over $100
  • Plan and values supplied$3.93bought at once in 2% · none losing over $100
Average loss per episode against the best available policy, Claude Opus 5.5 at low effort, about 99 episodes per condition on 100 held-out worlds. “Bought at once” means the agent committed at its first move instead of checking.
ChangeLoss per episode95% intervalReading
Plan first, against direct−$67.14−$92.3 to −$46.7Deliberating before acting removes most of the loss
Plan supplied, against its own plan−$0.66−$8.3 to +$6.8A better plan adds nothing detectable
Values supplied, against plan alone−$13.31−$18.9 to −$8.3Exact numbers close most of what remains
World-level paired differences, complete pairs (97 to 98 worlds). The first two were the declared tests, Holm-adjusted; the third is secondary.

What it shows

The agent could do the reasoning. Asked to plan, it wrote the best plan the format allows in 66 of 99 episodes, and handing it that plan did no better than its own. What it did not do, unprompted, was stop to reason before committing. Acting directly, it bought at once in nearly two episodes of three.

Even after planning, a gap remained between what the agent concluded and what it did. In 25 plan-first episodes and 34 plan-supplied episodes, its own estimate said checking was worth more than $5, and it bought at once anyway. Given the exact values, it did so twice. It acted on numbers it was handed far more reliably than on numbers it had worked out. We cannot yet say why: trust in an outside source, salience, the accuracy of its own estimates and its attitude to risk are not separated by this design.

None of this is visible in a completion rate. Every episode ends in a purchase. The loss sits in the order of moves: whether the agent looks before it commits.

What this does not show

  • One model, at one effort level, through one runtime.
  • The worlds were screened so that the supplied plan is close to the best policy; that is about 4% of the worlds the generator produces. In a smaller development set without that screen, the plan-first effect was not distinguishable from zero.
  • Seven of 400 episodes did not complete. Filling each with the loss of buying now, or with the largest loss observed in its condition, leaves both declared results standing; the worst case the worlds allow does not.
  • A simulated loss is not yet a real one. Whether these losses predict what an agent would cost a real buyer is the next question, and it needs real records to answer.

The implication

For anyone delegating purchases to an agent, the cheapest safeguard here was structural, not a better model or a better plan: require a written, valued plan before any irreversible commitment, and supply the numbers the agent should act on where they can be computed. For anyone evaluating agents, the lesson is to price decisions, not tasks. The same agent that completes every purchase can lose most of what the purchase was worth, at one identifiable step.