Two numbers, one job
Co-Design Bench asks a question closer to paid work than most coding tests do. The agent receives a running web application and a real user's change request, sometimes with an image, an audio clip, a video or a document attached as the brief. Hidden criteria, written by experts, then decide whether the revision did what was asked and left the rest of the app intact.
The strongest of ten tested endpoints averages 69.9% strict criterion completion per task, but fully passes only 26.4% of tasks. Those metrics answer different questions. The first averages each task's share of satisfied acceptance criteria; the second requires every criterion in a task to pass. The 73.6% of tasks that fail the full-pass test may be missing one requirement or many. The published aggregate figures do not reveal how much work remains on each failed task.
The Decoder identifies the strongest tested endpoint as GPT-6 Astra and reports a 41% full-pass rate on Multimodal Editing and approximately 18% on Repository Editing. Those track figures are attributed to The Decoder's reading of the results, rather than independently established by the desk from the paper abstract. The difference is substantial within this evaluation: localized, multimodal revisions were completed more often than coordinated repository edits. It does not establish that repository work is universally harder across coding tasks.
Built from sessions that already went wrong
The researchers began with 26,314 real coding sessions containing 37,463 deduplicated queries. They then selected benchmark tasks using difficulty, verifiability and historical failure signals, including evidence that users had rephrased requests or rolled back agent changes. The resulting 273 tasks are deliberately challenging, not a random sample of everyday coding work. Their failure rates should not be extrapolated to all coding-agent use.
Grading is strict in a specific way. Each criterion is judged pass, partial or fail against the running app, using source code and browser evidence. Partial earns no credit in the strict score. A separate verifier then challenges every positive verdict, looking for stubs, hardcoded outputs and features that exist but cannot be reached. A task counts as fully passed only when every criterion survives that process.
Repository-editing tasks involving three or four change types have lower strict criterion-completion scores than tasks involving one or two for all ten evaluated endpoints. The project reports a 6.3-percentage-point gap in this comparison. The association is consistent across the tested endpoints, but tasks with broader changes may also differ in complexity, length and other characteristics. The result does not isolate the causal cost of adding another change type.
Rankings also move between tracks: one smaller model, Qwen3.8-27B, climbs from fifth on multimodal editing to second on repository editing, which is a reminder that a single leaderboard number hides where a model is actually strong.
The execution profiles also show that greater effort does not necessarily correspond to better results across endpoints. The leading model uses a median of 32 turns on Multimodal Editing, while another tested endpoint uses 69 turns and approximately six times as many tokens yet scores lower. That comparison does not isolate the effect of additional turns: the models differ, and the study does not establish what would happen if the same endpoint were given a larger budget.
The execution environment matters too. The Decoder reports a separate comparison in which Qwen3.8-27B scored approximately 75% or 56% on repository-editing work depending on the agent harness. That result should not be confused with the shared-harness leaderboard: it illustrates how strongly the surrounding tools and execution setup can affect measured performance. A model's rank under one harness is not a universal ranking of coding-agent products.
What the 99.4% figure does and does not say
The project page also compares results on 150 WebCompass-edit tasks. For one model, switching from single-shot to agentic execution raises graded completion from 73.6% to 95.1%; a different model reaches 99.4% under agentic execution. These are results on a separate task pool and should not be presented as a direct comparison with Co-Design Bench's 26.4% full-pass rate.
The grading rules also differ. The cited WebCompass graded-completion measure gives partial credit, while Co-Design Bench's strict criterion score gives none and its full-pass measure requires every criterion to succeed. The task pools and execution protocols differ as well. A high graded-completion result on WebCompass therefore does not contradict a low full-task success rate on Co-Design Bench, nor does the comparison establish which benchmark better predicts production outcomes.
The benchmark has limits the reader should keep. Each task is a single fixed request, so it says nothing about the back-and-forth — a user correcting course mid-task — where real co-design actually happens. The judges are models, and the authors' judge-then-refute design reduces but does not remove grading error. Because the benchmark deliberately favors difficult, verifiable cases with historical failure signals, its 26.4% full-pass rate is not an estimate of how often coding agents finish ordinary user requests. Nor is it a statistical lower bound on that rate.
Sources
- [1] Co-Design Bench paper, arXiv:2610.35138 (abstract read first-hand, 8 November 2026)Read source
- [2] Co-Design Bench project page (read first-hand, 8 November 2026)Read source
- [3] The Decoder, "AI coding agents still need human cleanup on nearly three out of four tasks in a real-world test" (8 November 2026)Read source