Taste-Bench: Top AI Model Scores 59.7%, Chance Is 25%

The test end-to-end scores cannot run

Most agent benchmarks ask one question: did the task get finished? A finished task can still be a badly routed one, full of hours spent on a direction that looked reasonable at the time and was wrong. Taste-Bench, from a team at City University of Hong Kong and Microsoft, grades the routing instead.

The construction is the clever part. Agent systems routinely attempt the same task several times, and the attempts diverge. Where two attempts share the same history and then split, the recorded outcome of each branch says which direction was better. The researchers mine those splits, freeze the run at the split, hide everything after it, and present the two directions as a question. A second source of questions comes from single runs where the agent took a direction, hit a failure, and corrected itself: the test is whether the evaluated model can spot the mistake earlier than the agent that made it. The method avoids requiring manual grading of every fork, and in a human review of 100 sampled questions, reviewers agreed with the automatically mined labels on 170 of 172 explicit judgments (98.8%). That supports the labels' reliability in the reviewed sample; it does not establish that every benchmark label is correct.

One scoring choice shapes every number in the paper. Each question is asked twice, with the two candidates swapped, and it counts as correct only if the model is right both times. That removes position bias, and it resets the baseline: independent random guessing scores 25% in expectation, not the 50% of an ordinary two-choice test, and a model that always picks the same position scores 0%.

Not a coin flip — but not the whole story

Coverage of the paper in October 2026 has described the best model as barely beating a coin flip. Under the paper's rule that undersells the score and misses the actual problem. GPT-5.6 Sol averaged 59.7% across the research and engineering subsets — the paper's headline Average is the 1:1 mean of the two — against a 25% random-guessing baseline. GPT-5.5 was close behind at 59.5%, and the remaining twelve models spread out below them. The evaluated systems are models answering frozen decision questions, and the 59.7% is a macro-average of subset accuracies, not the share of all 502 questions answered correctly and not a real-world success rate.

The weakness sits further along the timeline. The researchers graded each fork by how far ahead a judge has to look before the better direction is clearly justified. Averaged across the 14 models evaluated, accuracy was 62.3% on forks where a decisive fact is already visible in the prefix, and fell to 21.0% on forks where the justification requires a completed check or substantial later work — below the protocol's 25% random-guessing baseline. Detour forks, the ones where the right move means abandoning a direction already taken, were harder than parallel forks in both domains.

Read that carefully. The paper measures how much subsequent work is needed before the better choice becomes clear; it does not measure what a wrong choice costs. The clearest weakness appears when choosing correctly requires information that emerges only after additional work — and those are also the decisions a supervisor cannot easily check in the moment, because the evidence has not happened yet.

More reasoning, within limits

The obvious repair is more reasoning at the fork. The paper tested it directly, rerunning the benchmark at three reasoning-effort settings for two models. Moving from the lowest to the highest setting changed GPT-5.6 Sol by −0.2 points and GPT-5.6 Luna by +2.2 points: limited, inconsistent effects, and the experiment does not establish that additional reasoning is ineffective for every model or task. The models were not coasting: at every setting they spent the most reasoning tokens on the hardest forks, and scored lowest there anyway. The deciding evidence at those forks exists only in work that has not happened yet, and no amount of staring at the prefix manufactures it.

Taste-Bench also separates models that end-to-end scores bunch together. Against SWE-bench Verified, the correlation is partial (r = +0.63 overall, +0.37 on the engineering subset), and the four models at the top of that leaderboard, within 4.0 points of each other there, sit 10.7 points apart here. Finishing tasks and choosing well inside them are related skills. They are not the same skill.

The hopeful half: judgment can be handed over

The paper's second half is a training result, and it is the part worth acting on. The team distilled the reasoning of a teacher that had been shown the correct direction into a student that sees only the question, using the benchmark's own forks as training material and holding out entire tasks. On unseen tasks the student's strict-protocol accuracy rose from 30.0% to 47.9%.

Then they let the student advise. Its fork judgments were written into the context of a separate executor agent running 41 held-out SWE-bench Pro tasks. Success rose from 14.6% with no advice to 33.7% with student advice — about 6 versus 14 of the 41 tasks — most of the way to the 39.0% achieved when every piece of advice was correct. That is encouraging evidence that learned fork-level guidance can help. It is also a small evaluation, confined to software engineering, and it does not establish reliable gains across arbitrary agent deployments.

For anyone running agents today, the usable implication is narrower than a product claim. The paper tested reasoning budgets and advice from a second model; it did not test human supervisors, and it does not compare the effectiveness or cost of human review against model advice. What its results support is experimenting with targeted review at consequential decision forks, rather than relying solely on end-of-task evaluation: human review and second-model advice are both plausible ways to supply the judgment the executing model lacks, and neither is validated by this paper over the other.

Sources

[1] Pan, W. et al., "The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks", arXiv:2609.25804, September 2026 — primary source, verified against the HTML v2 text: fork construction, the 502-question composition (390 engineering, 112 research), the both-orders scoring rule and its 25% random-guessing baseline, the 59.7% and 59.5% headline Averages, the 62.3% to 21.0% time-horizon fall as the mean over 14 models, the −0.2 and +2.2 reasoning-budget changes on the two models tested, the SWE-bench Verified comparison, the 170 of 172 human-label agreement, and the distillation and 41-task end-to-end resultsRead source
[2] Taste-Bench code repository — MIT-licensed code; dataset text CC BY 4.0; trajectories from SWE-bench, SWE-bench Pro and METR's MALT release, per the repositoryRead source
[3] AI Daily Post, "AI Agents Fail Long-Term Decision Tests, Study Shows", 18 October 2026 — secondary coverage inside the run window; its coin-flip framing of the 59.7% score does not reflect the paper's both-orders scoring rule and is corrected in this pieceRead source
[4] AI Parabellum, "AI Agents Still Struggle With Long-Term Decisions", 18 October 2026 — secondary coverage inside the run window; correctly notes the paper was first posted in September 2026Read source

THE ALFW VERDICT

Taste-Bench exposes a limitation that end-to-end benchmarks can obscure: choosing a promising direction before the outcome is known. The highest-scoring model averaged 59.7% across the benchmark's research and engineering subsets — well above the 25% random-guessing baseline its two-order scoring protocol sets — but averaged across the 14 models evaluated, accuracy deteriorated as the evidence needed to judge a decision moved further into the future, from 62.3% at the shortest horizon to 21.0% at the longest. Increasing reasoning effort produced limited, inconsistent gains in the two models tested. The encouraging result is that distilled decision guidance improved accuracy on held-out questions and raised task completion in a small 41-task engineering evaluation. The practical implication is to test targeted supervision and decision-point advice in long-running agents — not to assume that more reasoning tokens, or a strong end-to-end score, guarantee good judgment along the way.