Read the 25-point gain against the right baseline
The number doing the rounds is 39.2% to 64.7% on SWE-bench Pro. It is accurate, and the model card is unusually clear about the settings that produce it: the pi harness, thinking off, one attempt per task, the 51 hardest tasks in the benchmark. Under those settings the adapter newly solves 15 tasks and loses 2, and the card publishes the interval most releases omit: the paired gain is plus 25.5 points, plus or minus 14.2. On Terminal-Bench 2.1 the paired gain is 16.3 points, plus or minus 9.1, across 86 tasks.
The comparison that decides what this release actually is sits one table further down. Turn the base model's thinking on at its highest effort and it scores 64.7% on the same SWE-bench Pro set. The adapter does not beat the model's best setting. It removes the need to pay for it in generated tokens. Median generated tokens per task fall from 34.4k to 7.9k on SWE-bench Pro, and from 16.5k to 3.7k on Terminal-Bench. Median turns on tasks both versions solve fall from 59 to 23. For anyone serving a coding agent, that is the practical result: thinking-level accuracy at roughly a quarter of the generated tokens, from a downloadable adapter rather than a new model. On Terminal-Bench, the card's separate thinking comparison puts the adapter slightly ahead, 56.6% to the thinking base model's 54.2%, on a different task set from the paired table.
The card is equally direct about the reverse setting. Switch thinking on for the adapter and the runs show no clear improvement: the tasks only one version solves split 10 to 9 on SWE-bench Pro and 9 to 9 on Terminal-Bench. The card's own advice is to keep thinking off.
The caveats are printed on the same page
Three limits should travel with the headline. First, the sets are small. Fifty-one tasks is why the interval around the paired gain is 14 points wide; a handful of tasks moving the other way would change the story. Second, the checkpoint was not chosen blind. Step 160 was picked by looking at checkpoints' SWE-bench Pro scores and an 18-task Terminal-Bench subset during training, which makes the SWE-bench figure partly a selection result. The cleaner test is the one the card also runs: on the 68 Terminal-Bench tasks that played no part in picking the checkpoint, the gain against the thinking-off base is 16.2 points, close to the headline 16.3. That validates the gain over the base model. It does not independently validate the token savings or the parity with maximum-effort thinking, and it does not remove the selection issue on SWE-bench Pro.
Third, the result belongs to one harness. The adapter was trained inside pi, evaluated inside pi, and the card warns its numbers are not comparable with the base model's own card, which used other harnesses and the full benchmark. A sibling adapter exists for OpenCode, released the same way: the published pattern is one adapter per model-and-harness pair, not one adapter that travels. Anyone running a different agent should expect to train or wait for their own.
There is also a correction on the record. An earlier version of this card reported 29.4% to 62.7%. Nine of the base model's rollouts in that run had failed because the tunnel between the sandboxes and the model server went down mid-run. The team says it now checks every rollout for this and reran all the affected ones, on both sides. The correction raised both scores, and it shrank the claimed gain: the original table implied 33.3 points; the corrected one shows 25.5. Publishing a smaller gain than the first draft claimed is not the direction corrections usually move. It is still a reminder that agent benchmarks fail in mundane ways, and that the first version of any such table should be read as provisional.
What a local user can actually do with it
The adapter is a free Apache 2.0 download. It serves through vLLM with LoRA enabled alongside the base model, through transformers serve, through llama.cpp as a GGUF adapter on a Q8_0 base, or merged into a standalone checkpoint of about 54 GB in BF16. The expensive asset in this approach is the base model; the specialization is a small adapter file per workflow.
Two warnings from the card are worth keeping attached. The GGUF version has not been re-evaluated, and the further a quantization sits from the BF16 weights it was trained on, the further the behavior drifts. The 64.7% figure belongs to the evaluated configuration, not automatically to a smaller quant on a home machine. And the training run itself was short: the published checkpoint is step 160, from a run the card's training table says was stopped at step 191 of 1,000 planned.
Sources
- [1] Hugging Face, harness-hub/Qwen3.8-27B-pi-adapted model card, released November 18, 2026 — the primary source, read first-hand: paired benchmark tables with intervals and sign tests, the thinking comparison and token medians, checkpoint selection method, the held-out 68-task split, the correction note, training method and data, serving routes, license, and the GGUF not-re-evaluated warning.Read source
- [2] MarkTechPost, "New LoRA Adapter Lifts Qwen3.8 Coding Success From 39% to 65%", November 18, 2026 — secondary report of the same release; an independent restatement of the card's tables.Read source