Guide
LSCBench: the best coding agent passes 91% of tests but only 25% are safe to ship
Six frontier coding agents, 450 tasks built from 150 real vulnerabilities, and a specification that never mentions security. Functional pass rates reach 91.3%; the best score for code that both works and stops the original attack is 25.1%. Here is what the benchmark actually measured, where agent-written code breaks, and the cheapest fix the authors tested.
- Published
- 19 Oct 2026
- Reading
- 8 min
- Class
- guide
half-life 90dfrom 19 Oct 2026
On 10 October 2026, researchers at the National University of Singapore and Alibaba Group — the author list includes members of the Qwen team — posted a benchmark to arXiv that turns a hunch into a number. Across six frontier coding agents and 450 tasks built from 150 real vulnerabilities, functional pass rates ran from 73.6% to 91.3%. The best score for code that both does the job and resists the original attack was 25.1% (Last Secure Code Benchmark, arXiv:2610.14766, submitted 10 October 2026).
The gap between those two numbers is the whole point. It lands in a year when AI-written code is already ordinary: Stack Overflow's 2025 developer survey found 81% of developers were concerned about the security and privacy of AI tools, and a Harvard study of 718 firms published this month showed teams writing 30% more code without shipping more features (The review bottleneck, 12 October 2026). LSCBench does not measure opinion. It fires the real exploit at what the agent produced and records whether it still works.
What the benchmark tests, and how it scores
Every task starts from a real project with known-vulnerable code removed, plus a specification of what to rebuild. The specification never mentions security — every requirement holds for both the vulnerable and the fixed version of the code, so nothing in the prompt hints that there is a hole to close. Recognising the risk is left to the agent, exactly as it is in everyday use (project page, read 19 October 2026).
The 150 vulnerability scenarios each appear at three sizes. Function masks the code region where the vulnerability lived and leaves the rest of the project intact. File deletes the file that contained it, so the agent writes that file from a specification. Repository empties the source tree — the whole repository when it is under 8,000 lines, otherwise the files around the patch, up to 8,000 lines — and the agent rebuilds it from a single specification. That makes 450 tasks (README, checked 19 October 2026).
The sources are real: 115 scenarios come from public CVEs and 35 are Java applications written by security experts. The set covers seven languages (C, C++, Go, Java, JavaScript, PHP and Python) and 24 CWE categories, including 21 of the 2025 CWE Top 25 and 8 of the top 10 known-exploited weaknesses — more coverage, the authors argue, than any benchmark they compared against.
Grading happens after the agent stops, in a separate container, with two hidden suites: does the code do what the specification asked, and does the original attack still get through? A task counts only when both are true. There is no partial credit and no judge model deciding how close an attempt came. The agents ran as people actually run them — five inside Claude Code v2.1.143, GPT-5.6-Sol inside Codex v0.154.0, at their highest reasoning effort, one attempt per task (README, checked 19 October 2026).
The leaderboard, and the row that should worry you
| Model | Agent | Works | Works and secure |
|---|---|---|---|
| Claude Fable 5 | Claude Code | 91.1% | 25.1% |
| GPT-5.6-Sol | Codex | 89.1% | 24.7% |
| Qwen3.8-Max | Claude Code | 89.3% | 22.0% |
| GLM-5.3 | Claude Code | 87.1% | 21.3% |
| DeepSeek-V4-Pro | Claude Code | 73.6% | 18.0% |
| Kimi-K3 | Claude Code | 91.3% | 16.4% |
Share of the 450 tasks each model solved, both at once in the last column. Source: last-secure-code-benchmark.github.io, read 19 October 2026.
Read the bottom row first. Kimi-K3 has the highest functional pass rate in the field at 91.3% and the lowest combined score at 16.4%. A green test suite told you nothing about which of these agents was safer to use. Claude Fable 5 tops the combined ranking, and even it leaves roughly three in four completed tasks exploitable.
The aggregates are worse than any single row. Across all 2,700 runs — six models on 450 tasks each — 66% ended with code that works and is still vulnerable. Of the 150 scenarios, 57 were never secured by any model, not once. Only 26 were secured by all six models at one granularity or more. The paper's own summary: the best model takes a scenario through both suites at all three of those sizes on just 12% of the vulnerability scenarios.
Where agent-written code actually breaks
Grouped by language, runs that ended both working and secure range from 30% in C and Python down to 10% in PHP, with JavaScript at 27%, C++ at 19%, Java 18% and Go 14%. Grouped by weakness family — measured as the share of working code that was also secure — the spread is wider: memory safety 44%, injection 28%, path and file handling 27%, access control 20%, deserialization 20%, input validation 17%, server-side request forgery 3%, and information exposure 1%.
Those last two are not exotic bugs; they are decisions about what to withhold. The
paper's case studies show the pattern. On a Java configuration importer task, Kimi-K3
used SnakeYAML's default constructor, which honours every !!class tag in
an uploaded file; Claude Fable 5 and GPT-5.6-Sol used SafeConstructor
instead, and Qwen3.8-Max and GLM-5.3 made the same mistake as Kimi-K3. On a LocalAI
task in Go, no model confined URL reads to the models directory, so file://
paths went through. On a syft task, all six models followed the tag convention of
neighbouring structs, which put an attestation password into config dumps and SBOM
output (arXiv HTML, case studies, read 19 October 2026).
One finding cuts against a common assumption about long context. From function-level to repository-level tasks, the average functional pass rate falls from 93% to 80%, while the combined working-and-secure score stays between 19% and 23%. Giving the agent more of the codebase costs you correctness, not security. The security gap is not a context problem.
One sentence helps. It does not fix it.
The authors reran four of the six models with a single line appended to every task, asking for an implementation that meets the functional requirements “while also being secure”. Works-and-secure scores moved as follows: GPT-5.6-Sol 24.7% to 46.2%, GLM-5.3 21.3% to 38.2%, Qwen3.8-Max 22.0% to 37.6%, and DeepSeek-V4-Pro 18.0% to 19.1%. Functional pass rates shifted by a few points either way. Claude Fable 5 and Kimi-K3 were not part of that rerun (arXiv:2610.14766, Section 5.5, read 19 October 2026).
Three of the four roughly doubled their secure output from one sentence. The paper's verdict is still that most tasks stay insecure even with the reminder, and that server-side request forgery and information exposure stay near the bottom. Asking helps; it is not a control.
What to do with this on Monday
- Put the sentence where your agent reads it. The paper
appended it to the task prompt. The everyday equivalent is a line in
AGENTS.mdorCLAUDE.md— the paper did not test that placement, so treat it as unproven but cheap: prefer the safe API when a library offers one, and stop and ask when a requirement conflicts with security. - Write the second suite. The benchmark's security test replays the original attack rather than checking outputs look right. Copy the shape: if your agent built a loader that takes a model name, assert that it refuses a URL and a path that climbs out of the directory, then run that assertion in CI alongside your functional tests. If nobody wrote the adversarial test, the working-and-vulnerable case is invisible by construction.
- Run the benchmark on your own stack. LSCBench ships as a Harbor dataset under Apache 2.0, and needs Harbor 0.24 or later, Python 3.12 or later and Docker; the task images run to about 58 GB (README and dev.to, read 19 October 2026). Start with the oracle sanity check so the reference solutions score 1, and run it in an isolated environment — the task projects contain live vulnerabilities on purpose.
- Review in small pieces, and review for security on purpose. Review time is already the bottleneck in most teams (the 718-firm study), and a tests-only gate will happily pass the 66% case. Grade agent work the way this benchmark does: does it work, and does the attack still land.
What the benchmark does not prove
Four limits worth holding onto. It is a 10-page arXiv preprint posted on 10 October 2026, not a peer-reviewed paper. It tests the model versions available when it was written, at pass@1 — a single attempt per task — so current releases may behave differently, which is an argument for re-running it yourself rather than quoting the table forever. Qwen3.8-Max was both a scored model and part of the team that built the tests; the authors argue its third-place result at 22.0% shows no advantage from that involvement, and you may weigh that differently. And the harnesses are pinned — Claude Code v2.1.143 and Codex v0.154.0 — so a large part of any future movement will come from harness changes, not model changes.
None of that weakens the finding that matters. Functional tests answer “does it do what I asked”. LSCBench asked the other half — “does it still open where the last version opened” — and no agent cleared a third of it. Expect “it passes our tests” to stop being a sufficient merge argument, and start writing the suite that asks the second question (Evals that actually catch regressions). The paper's own conclusion is the shortest version: this code should not be deployed in the real world without careful review.
Sources
- Last Secure Code Benchmark: Is the Code You Vibe Secure? (arXiv:2610.14766) — submitted 10 October 2026, National University of Singapore and Alibaba Group. Primary source: benchmark design, task counts, CWE and KEV coverage, headline functional and functional-and-secure rates, the reminder experiment (Section 5.5) and the case studies.
- LSCBench project page — read 19 October 2026. Leaderboard figures, the 66% of 2,700 runs, 57 of 150 scenarios never secured, language and weakness-family breakdowns, the context-tier finding.
- alibaba/last-secure-code-benchmark README — checked 19 October 2026. Granularity rules, harness versions (Claude Code v2.1.143, Codex v0.154.0), Harbor requirements, Apache 2.0 licence, the oracle sanity check.
- dev.to: Coding agents pass up to 91% of tasks — at most 25% are also secure — read 19 October 2026. Independent reading of the leaderboard, Section 5.5 reminder results, task-image size and run instructions.
- Tech Insider: AI coding agents pass 91% of tests, just 25% are secure — 16 October 2026. Coverage of the release and its reception.
- Stack Overflow Developer Survey 2025: AI — 2025. Developer concern about AI tool security and privacy (81%).
- The review bottleneck: Harvard's 718-firm study — 12 October 2026, fullauto.online. The throughput side of the same problem.