---
title: "Systematic Capability Benchmarking of Frontier Large Language Models for Offensive Cyber Tasks (arXiv:2604.17159) | Papers With Code Cyber"
description: "We present, to our knowledge, the most comprehensive cross-model evaluation of LLM agents on offensive cybersecurity tasks, benchmarking 10 frontier models from 7 providers…"
canonical: "https://paperswithcode-cyber.com/paper/2604.17159"
type: "paper"
---

# Systematic Capability Benchmarking of Frontier Large Language Models for Offensive Cyber Tasks

arXiv:2604\.17159 · 2026-04-18

Authors: Tyler H\. Merves, Michael H\. Conaway, Joseph M\. Escobar, Hakan T\. Otal, Unal Tatar

## Abstract

We present, to our knowledge, the most comprehensive cross\-model evaluation of LLM agents on offensive cybersecurity tasks, benchmarking 10 frontier models from 7 providers on all 200 challenges of the NYU CTF Bench\. Building on the D\-CIPHER multi\-agent framework, we extend it with multi\-provider backend support, a custom Kali Linux environment with over 100 pre\-installed penetration testing tools, and runtime tool\-discovery agents\. Through a controlled factorial study, we find that the Kali Linux environment yields a \+9\.5 percentage\-point improvement over Ubuntu, while auto\-prompting and category\-specific tips often degrade performance in well\-equipped environments\. Among models, Claude 4\.5 Opus achieves the highest solve rate \(59%\), followed by Gemini 3 Pro \(52%\), with Gemini 3 Flash offering the best cost\-efficiency at $0\.05 per solve\. Asymmetric planner/executor model assignments provide no meaningful benefit while coherent same\-model configurations consistently outperform mixed\-tier pairings\. Our results indicate that environment tooling and model selection emerge as the strongest drivers of performance, whereas prompt engineering interventions show diminishing or negative returns in well\-equipped environments\. Reported performance reflects both model reasoning ability and compatibility with agent tooling and API integration\.

## Tasks

- [Penetration testing](https://paperswithcode-cyber.com/tasks/cyberintel-penetration-testing)

## Code repositories

- [TATAR\-LAB/ctf\-agents](https://github.com/TATAR-LAB/ctf-agents)

## Paper resources

- [Abstract page](https://arxiv.org/abs/2604.17159v1)
- [PDF](https://arxiv.org/pdf/2604.17159v1.pdf)