---
title: "Cybersecurity AI Benchmark (CAIBench): A Meta-Benchmark for Evaluating Cybersecurity AI Agents (arXiv:2510.24317) | Papers With Code Cyber"
description: "Cybersecurity spans multiple interconnected domains, complicating the development of meaningful, labor-relevant benchmarks. Existing benchmarks assess isolated skills…"
canonical: "https://paperswithcode-cyber.com/paper/2510.24317"
type: "paper"
---

# Cybersecurity AI Benchmark \(CAIBench\): A Meta\-Benchmark for Evaluating Cybersecurity AI Agents

arXiv:2510\.24317 · 2025-10-28

Authors: María Sanz\-Gómez, Víctor Mayoral\-Vilches, Francesco Balassone, Luis Javier Navarrete\-Lozano, Cristóbal R\. J\. Veas Chavez, Maite del Mundo de Torres

## Abstract

Cybersecurity spans multiple interconnected domains, complicating the development of meaningful, labor\-relevant benchmarks\. Existing benchmarks assess isolated skills rather than integrated performance\. We find that pre\-trained knowledge of cybersecurity in LLMs does not imply attack and defense abilities, revealing a gap between knowledge and capability\. To address this limitation, we present the Cybersecurity AI Benchmark \(CAIBench\), a modular meta\-benchmark framework that allows evaluating LLM models and agents across offensive and defensive cybersecurity domains, taking a step towards meaningfully measuring their labor\-relevance\. CAIBench integrates five evaluation categories, covering over 10,000 instances: Jeopardy\-style CTFs, Attack and Defense CTFs, Cyber Range exercises, knowledge benchmarks, and privacy assessments\. Key novel contributions include systematic simultaneous offensive\-defensive evaluation, robotics\-focused cybersecurity challenges \(RCTF2\), and privacy\-preserving performance assessment \(CyberPII\-Bench\)\. Evaluation of state\-of\-the\-art AI models reveals saturation on security knowledge metrics \(\~70\\% success\) but substantial degradation in multi\-step adversarial \(A\\&D\) scenarios \(20\-40\\% success\), or worse in robotic targets \(22\\% success\)\. The combination of framework scaffolding and LLM model choice significantly impacts performance; we find that proper matches improve up to 2\.6$\\times$ variance in Attack and Defense CTFs\. These results demonstrate a pronounced gap between conceptual knowledge and adaptive capability, emphasizing the need for a meta\-benchmark\.

## Tasks

- [Penetration testing](https://paperswithcode-cyber.com/tasks/cyberintel-penetration-testing)

## Code repositories

- [aliasrobotics/cai](https://github.com/aliasrobotics/cai)

## Paper resources

- [Abstract page](https://arxiv.org/abs/2510.24317v1)
- [PDF](https://arxiv.org/pdf/2510.24317v1.pdf)