DeepSeek Scores 72.8%, Gated GPT-6 Sol Hits 80.8%

A test built like a break-in, not a quiz

Most offensive-security benchmarks hand a model a single puzzle. Vals built two small companies instead. Each agent starts on its own workstation with the address of one public web application and an authorized scope, and has to work the whole chain itself: exploit the application, gain administrator rights, map the network, collect and reuse passwords, and move between machines and domains. Lighthouse adds a second company that can only be reached with information stolen from the first, plus a simulated phishing step.

Scoring is the part that makes the numbers worth reading. An objective counts only when a hidden marker shows up in the recorded command output. The model cannot talk its way to a point, and Vals says it reviewed every run and found no shortcuts: each credited objective traces to a machine the agent had actually compromised.

Where the gap is, and where it is not

On Nova, the two models produced similar results. According to The Decoder's reporting of Vals' data, DeepSeek averaged 16.3 of 21 objectives across three runs, compared with 15.7 for Daybreak. Their observed ranges overlapped substantially: 15–18 objectives for DeepSeek and 12–19 for Daybreak. DeepSeek's numerical lead is small relative to that variation. The published figures do not establish a statistically reliable advantage for either model on Nova.

The overall score gives Nova and Lighthouse equal weight, even though Nova has 21 objectives and Lighthouse has 31. The reported eight-percentage-point difference therefore measures the gap between two equally weighted range scores, not a difference in the percentage of all objectives completed across a single combined pool.

The larger reported difference appears on Lighthouse. Daybreak completed 27 of 31 objectives in under three hours, while DeepSeek completed 21. That is a six-objective advantage on the more complex, two-company range. Vals' results also show Daybreak making faster progress with fewer tokens. The comparison supports a performance advantage in this benchmark, not a general conclusion that restricted access itself produces better or more reliable attack agents.

Vals reports that Daybreak reached 25 of Lighthouse's 31 objectives within 100 million cumulative tokens. DeepSeek completed no objectives during its first 25 million tokens and made much of its progress after 100 million. Vals characterizes DeepSeek's runs as consuming roughly two to four times as many tokens. These figures illustrate different progress and resource-use patterns, but the models did not complete identical sets of objectives, so the token comparison is not a controlled cost-per-objective measurement. Vals argues that tight early compute caps can materially distort rankings of models that progress more slowly.

The limits Vals prints on its own result

No model finished. The final Lighthouse objectives, which require carrying stolen access through to the last target, were completed by nobody, and only one run in the entire set reached Nova's final stage. Vals reports that every run terminated before reaching the 500-million-token or 48-hour limit. The scores therefore were not directly truncated by those formal caps. That does not establish why each agent stopped or whether additional prompting, a different harness or a changed strategy would have improved its result. The weakest entry makes the point vividly: GLM 5.3 Flash stalled at five objectives per range, and on one range spent roughly 36 hours and a thousand turns, much of it, Vals' execution traces show, sleeping and polling for a background job that never returned.

Two further discounts apply. The networks are simulated, with planted weaknesses and no defending team, so neither score says anything about success rates against a real, watched network. And Vals is the only source for Vals' numbers: no independent lab has replicated the runs, and the Nova figures are three-run averages with wide ranges. The eight-point headline gap should be read with that spread in mind.

Separate research provides context, not replication. According to The Decoder's account of UK AI Security Institute findings, the estimated open-versus-closed cyber-capability gap narrowed to four to seven months in July 2026, from six to ten months for much of 2025. The same reporting cites a DeepSeek V4-Pro run costing $1.19 versus $85 for Anthropic's Opus 4.5 on a different range. Those figures involve different models and evaluation conditions; they should not be used to validate Vals' eight-point gap or its $3-versus-$22 cost comparison.

Vals argues that increasingly capable open-weight models could help defenders run their own security assessments while also lowering access barriers for attackers. Its expectation of more real-world attacks is a forecast, not an observed outcome of the benchmark.

Sources

[1] Vals AI, CyberOffense Leaderboard and Methodology (primary)Read source
[2] The Decoder, "The gap between open and closed AI models in cyber capabilities keeps shrinking", 7 November 2026Read source

THE ALFW VERDICT

Vals' CyberOffense results show how close a downloadable open-weight model can come to a restricted frontier model on two simulated intrusion ranges. DeepSeek V4.1 Flash scored 72.8% against GPT-6 Sol Daybreak's 80.8%, with similar three-run averages on Nova but a larger deficit on Lighthouse. Vals also reports substantially lower run costs for DeepSeek. The comparison does not show that open and restricted models are equivalent, or that the reported costs apply to every deployment. The networks are simulated, the overall score averages two range percentages equally, and the published results have not been independently reproduced. For defenders, the finding supports evaluating open-weight agents in authorized security testing. For policymakers and model providers, it illustrates the limits of access restrictions on a single model when other capable models can be downloaded. Neither conclusion requires treating benchmark completion as a real-world breach probability.