---
title: "CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale (arXiv:2506.02548) | Papers With Code Cyber"
description: "AI agents have significant potential to reshape cybersecurity, making a thorough assessment of their capabilities critical. However, existing evaluations fall short…"
canonical: "https://paperswithcode-cyber.com/paper/2506.02548"
type: "paper"
---

# CyberGym: Evaluating AI Agents' Real\-World Cybersecurity Capabilities at Scale

arXiv:2506\.02548 · 2025-06-03

Authors: Wang, Zhun, Shi, Tianneng, He, Jingxuan, Cai, Matthew, Zhang, Jialin, Song, Dawn

## Abstract

AI agents have significant potential to reshape cybersecurity, making a thorough assessment of their capabilities critical\. However, existing evaluations fall short, because they are based on small\-scale benchmarks and only measure static outcomes, failing to capture the full, dynamic range of real\-world security challenges\. To address these limitations, we introduce CyberGym, a large\-scale benchmark featuring 1,507 real\-world vulnerabilities across 188 software projects\. Adjustable to different vulnerability analysis settings, CyberGym primarily tasks agents with generating a proof\-of\-concept test that reproduces a vulnerability, given only its text description and the corresponding codebase\. Our extensive evaluation highlights that CyberGym effectively differentiates agents' and models' cybersecurity capabilities\. Even the top\-performing combinations only achieve a \~20% success rate, demonstrating the overall difficulty of CyberGym\. Beyond static benchmarking, we show that CyberGym leads to the discovery of 34 zero\-day vulnerabilities and 18 historically incomplete patches\. These results underscore that CyberGym is not only a robust benchmark for measuring AI's progress in cybersecurity but also a platform for creating direct, real\-world security impact\.

## Tasks

- [Exploit construction and validation](https://paperswithcode-cyber.com/tasks/cyberintel-exploit-development)
- [Vulnerability detection and triage](https://paperswithcode-cyber.com/tasks/cyberintel-vulnerability-analysis)

## Code repositories

- [sunblaze\-ucb/cybergym](https://github.com/sunblaze-ucb/cybergym)

## Paper resources

- [Abstract page](https://arxiv.org/abs/2506.02548v3)
- [PDF](https://arxiv.org/pdf/2506.02548v3.pdf)