---
title: "SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks? (arXiv:2605.26548) | Papers With Code Cyber"
description: "Finding a real vulnerability in complicated systems is a challenging, long-horizon task that demands reasoning across an entire codebase to produce a working…"
canonical: "https://paperswithcode-cyber.com/paper/2605.26548"
type: "paper"
---

# SEC\-bench Pro: Can Language Models Solve Long\-Horizon Software Security Tasks?

arXiv:2605\.26548 · 2026-05-26

Authors: Hwiwon Lee, Jiawei Liu, Dongjun Kim, Wubing Xia, Ziqi Zhang, Chunqiu Steven Xia, Lingming Zhang

## Abstract

Finding a real vulnerability in complicated systems is a challenging, long\-horizon task that demands reasoning across an entire codebase to produce a working proof\-of\-concept \(PoC\)\. However, such critical security problems remain understudied\. We present SEC\-bench Pro, a benchmark that measures how well frontier models hunt real vulnerabilities by reproducing working PoC inputs from disclosed reports, where each task pairs a concrete bug with the instructions for triggering it\. We also demonstrate the limitations of existing rule\-based judges for grading generated PoCs, and propose a novel LLM\-based judge for more precise grading\. We instantiate SEC\-bench Pro with 344 validated vulnerabilities across three targets, the V8 and SpiderMonkey browser engines and the Linux kernel, covering critical vulnerability families including memory\-safety, sandbox, JIT, race\-condition, and kernel\-subsystem bugs\. Across six frontier commercial and open\-weight models and three coding agents, the strongest, Codex with GPT\-5\.5, solves 58% of instances overall\. We also observe that Claude Code with Opus 4\.6 tends to time out but solves most instances it completes\. In contrast, open\-weight models struggle; for example, GLM\-5 solves only 13 of the 344 instances\. During construction and evaluation, SEC\-bench Pro also surfaced three vulnerabilities in V8 and SpiderMonkey, including a sandbox escape that was fixed and earned a $20,000 Google Vulnerability Reward Program bounty\. More recently, SEC\-bench Pro has been adopted by OpenAI to evaluate the long\-horizon security capabilities of its newest models\. Overall, SEC\-bench Pro exposes where long\-horizon vulnerability discovery succeeds, where it fails, and how different grading choices change the evaluation landscape, offering insights for security\-centric model evaluation and training\. Our artifact is available at https://github\.com/SEC\-bench/SEC\-bench\-Pro\.

## Tasks

- [Exploit construction and validation](https://paperswithcode-cyber.com/tasks/cyberintel-exploit-development)

## Code repositories

- [SEC\-bench/SEC\-bench\-Pro](https://github.com/SEC-bench/SEC-bench-Pro)

## Paper resources

- [Abstract page](https://arxiv.org/abs/2605.26548v2)
- [PDF](https://arxiv.org/pdf/2605.26548v2.pdf)