Agentic Binary Reverse-Engineering Capability Gaps

Description

SRE-Bench evaluates whether cybersecurity agents can recover the behavior of realistic binary-only software without relying on source-code memorization. The authors construct 19 private programs, 44 anti-analysis primitives, 262 binary instances, and 1,572 deterministic grading tasks covering security-relevant reverse-engineering scenarios.

Examples

Use the authors’ controlled private-program benchmark, isolated binary-analysis environments, deterministic grading, fixed tool budgets, and nonproduction artifacts; see https://arxiv.org/html/2608.11469 (opens in a new tab).

Impact

GPT-5.6-sol achieves 61.4% mean rubric credit and fully solves 80 of 254 gradeable instances (31.5%); Claude-Opus-5 achieves 31.8% and fully solves 32 of 256 gradeable instances (12.5%). Those denominators exclude refusals and context-window failures, and every run is limited to 500 agent steps and six hours. The study measures benchmark capability gaps, not an exploitable production vulnerability or compromise rate.

Affected Systems

• mini-SWE-agent reverse-engineering harness with the paper’s restricted binary-analysis environment
• Automated malware-analysis, firmware-inspection, binary triage, and proprietary-software investigation workflows
• Security-analysis agents processing compiled programs without access to their original source code

Mitigation Steps

• Evaluate binary-analysis agents on withheld, independently authored programs with deterministic graders rather than relying solely on public-source benchmarks.
• Report refusal, context-exhaustion, and timeout rates separately from gradeable-instance scores.
• Use defense-in-depth and expert human review for security conclusions drawn from incomplete binary analysis.
• Isolate untrusted binaries, constrain execution privileges, and retain inspectable tool and grading traces.
• Measure performance across anti-analysis protections, program domains, and realistic execution budgets before extrapolating to production incidents.

Evidence

• https://arxiv.org/html/2608.11469 (opens in a new tab) — Abstract and Section 3 describe 19 privately authored programs, 44 anti-analysis primitives, 262 binaries, and 1,572 deterministic tasks.
• https://arxiv.org/html/2608.11469 (opens in a new tab) — Section 4 directly identifies GPT-5.6-sol, Claude-Opus-5, GPT-5.5, Grok-4.5, and GLM-5.2 under the same mini-SWE-agent harness.
• https://arxiv.org/html/2608.11469 (opens in a new tab) — Table 4 reports 61.4% and 80/254 fully solved for GPT-5.6-sol, versus 31.8% and 32/256 for Claude-Opus-5.
• https://arxiv.org/html/2608.11469 (opens in a new tab) — Section 4 limits runs to 500 steps and six hours and explicitly excludes refusals and context-window failures from reported averages.