A test with nothing to copy from

Most coding benchmarks hand a model an existing codebase and ask for a fix. ProgramBench removes the codebase. The agent gets a running program and its manual, and has to work out what the program does by running it, then design and write a replacement that behaves the same way on every test the graders can generate. It picks the language. It designs the architecture. If its replacement misses an edge case in how, say, jq formats an error, that task is not resolved.

The restrictions are the point. The binary is execute-only, so decompilers, disassemblers and even strings fail against it. There is no internet, because in early trials without restrictions, the benchmark's authors report, models found shortcuts such as cloning the original repositories from GitHub. What remains is observation and construction, which is why the launch result was so stark: in May, nine frontier models, 200 programs, not one task fully resolved. The paper adds a detail worth keeping: the models tended to write monolithic, single-file implementations, a style that diverges from how the original programs were built.

Two columns, two different stories

The headline column moved from 0% to 18.5% in under six months, and that movement is corroborated. Vals, running the same benchmark under its own setup (the same harness, a 1,000-step limit and a six-hour wall clock), also counts 37 fully resolved tasks for Opus 5.5. Agreement across two evaluation runs strengthens confidence in that result, although both use ProgramBench's task set and grading criteria.

The second column is where the agreement ends. Almost resolved counts a task when at least 95% of its tests pass. The official board puts Opus 5.5 at 141 such tasks; Vals counts 130. Same model, same benchmark, an eleven-task gap on the generous metric, and none on the strict one. The benchmark's authors anticipated this use of the number. Their documentation says the leaderboard's primary metric is fully resolved, that almost resolved is reported as orientation while that metric is low, and that a single failed test can indicate a severe problem in a program. On their own reasoning, a rebuild passing 95% of tests on a large utility can still be a program you would not ship.

Vals' raw pass rates show why the strict column separates models that otherwise look tied. Opus 5.5 passes 87.0% of all tests on average; GPT-6 Astra passes 85.4%. Read as averages, they are close. Read as complete programs, the counts are 37 against 11 on Vals' board, and 37 against 10 on the official one. Passing most tests is different from satisfying every test in a task's evaluation suite. The fully resolved metric captures that stricter threshold, although even a perfect benchmark result does not establish that a replacement is production-ready or equivalent on every possible input.

What 37 out of 200 does not settle

The remaining 163 tasks include the programs that define the hard end of the test. On the official leaderboard, no model's best attempt on FFmpeg passes more than 14% of its tests. The PHP interpreter's best is 33%, DuckDB's 53%. The fully resolved 37 are concentrated where you would expect: smaller, well-specified command-line tools, several of which at least one model now rebuilds completely. Whole large systems remain out of reach, and the cost of the leading run, $36.36 per task on average, buys a one-in-five completion rate under test conditions, with no maintenance, security review or integration work counted.

Clark's expectation that the benchmark saturates by the end of 2027 may prove right; the slope from May to October is steep. That is a forecast, not a trajectory established by two leaderboard snapshots. But saturation talk should be attached to the strict column, where the score is 18.5%, not to the orientation column, where a model can look two-thirds done while completing fewer than one program in five. The next update worth reporting is simple: whether the fully resolved count keeps climbing at this pace, and whether anything at the FFmpeg end of the table starts to move.

Sources

  1. [1] ProgramBench, official leaderboard and FAQ (primary source: current standings for Claude Opus 5.5, Claude Fable 5.1 and GPT-6 Astra, per-task best scores, the statement that fully resolved is the primary metric to report, and the warnings on average pass rates and almost resolved)Read source
  2. [2] Yang et al., "ProgramBench: Can Language Models Rebuild Programs From Scratch?", arXiv, May 6, 2026 (primary source: benchmark design, 200 tasks, nine models at launch, 0% fully resolved, best almost resolved 3.0% for Claude Opus 4.7, monolithic single-file tendency)Read source
  3. [3] Vals AI, ProgramBench leaderboard and methodology (independent evaluation, page updated October 20, 2026: Opus 5.5 at 37 fully resolved and 130 almost resolved, raw pass rates, harness and limits)Read source
  4. [4] Jack Clark, Import AI 478, October 26, 2026 (the newsletter that reported the updated leaderboard and stated the expectation of saturation by the end of 2027)Read source