Repository-Level Coding Agent Vulnerability Detection Gaps

Description

VulnGym measures whether coding agents can locate and explain repository-level security vulnerabilities from realistic advisory and source-code context. The benchmark contains 184 reviewed advisories, 408 line-annotated vulnerability entries, and 23 repositories, with separate end-to-end detection and oracle-conditioned localization tasks.

Examples

Reproduce defensively against fixed benchmark repository snapshots using read-only file inspection, known advisory labels, line-level grading, and no exploit execution; see https://arxiv.org/html/2608.02001 (opens in a new tab).

Impact

On the benchmark’s hard subset, the strongest tested agent-model combination, OpenHands with DeepSeek-V4-Flash, reaches 22.58% advisory recall, 15.22% entry recall, and 11.63% edit similarity. The findings describe missed benchmark vulnerabilities, not proof that a deployed scanner always misses that fraction of real issues. The reported benchmark results are limited by repository selection, advisory context, line-matching tolerance, read-only tool restrictions, and oracle-conditioned subtasks.

Affected Systems

• Claude Code, OpenHands, and MiniSWE coding-agent scaffolds under the paper’s evaluated configurations
• Repository-level code-review, security-triage, advisory-localization, and vulnerability-trace workflows
• Security analysis agents restricted to read-only repository-search and file-inspection commands

Mitigation Steps

• Evaluate repository-level security tools against reviewed vulnerabilities with line-level labels and reproducible repository snapshots.
• Report advisory recall, individual-entry recall, localization quality, and false positives as separate metrics.
• Pair agent findings with static analysis, independent review, and targeted regression tests before asserting coverage.
• Improve cross-file trace construction and security-sensitive data-flow reasoning rather than relying on isolated file summaries.
• Retain expert review for high-risk repositories and treat oracle-conditioned subtask scores separately from end-to-end discovery.

Evidence

• https://arxiv.org/html/2608.02001 (opens in a new tab) — Benchmark construction specifies 184 reviewed advisories, 408 line-annotated vulnerability entries, and 23 repositories.
• https://arxiv.org/html/2608.02001 (opens in a new tab) — Experimental Setup identifies DeepSeek-V4-Flash, GLM-5.2, MiniMax-M3, and Qwen3.5 at 2B, 4B, 9B, and 27B.
• https://arxiv.org/html/2608.02001 (opens in a new tab) — Table 2 reports OpenHands with DeepSeek-V4-Flash at 22.58% advisory recall, 15.22% entry recall, and 11.63% edit similarity on hard cases.
• https://arxiv.org/html/2608.02001 (opens in a new tab) — Experimental Setup distinguishes Claude Code, OpenHands, and MiniSWE as agent scaffolds and restricts agents to read-only repository commands.