---
title: "CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Models (arXiv:2408.01605) | Papers With Code Cyber"
description: "We are releasing a new suite of security benchmarks for LLMs, CYBERSECEVAL 3, to continue the conversation on empirically measuring LLM cybersecurity risks and…"
canonical: "https://paperswithcode-cyber.com/paper/2408.01605"
type: "paper"
---

# CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Models

arXiv:2408\.01605 · 2024-08-02

Authors: Wan, Shengye, Nikolaidis, Cyrus, Song, Daniel, Molnar, David, Crnkovich, James, Grace, Jayson, Bhatt, Manish, Chennabasappa, Sahana, Whitman, Spencer, Ding, Stephanie, Ionescu, Vlad, Li, Yue, Saxe, Joshua

## Abstract

We are releasing a new suite of security benchmarks for LLMs, CYBERSECEVAL 3, to continue the conversation on empirically measuring LLM cybersecurity risks and capabilities\. CYBERSECEVAL 3 assesses 8 different risks across two broad categories: risk to third parties, and risk to application developers and end users\. Compared to previous work, we add new areas focused on offensive security capabilities: automated social engineering, scaling manual offensive cyber operations, and autonomous offensive cyber operations\. In this paper we discuss applying these benchmarks to the Llama 3 models and a suite of contemporaneous state\-of\-the\-art LLMs, enabling us to contextualize risks both with and without mitigations in place\.

## Tasks

- [AI\-system security](https://paperswithcode-cyber.com/tasks/cyberintel-ai-system-security)
- [Penetration testing](https://paperswithcode-cyber.com/tasks/cyberintel-penetration-testing)

## Organizations

- [Meta](https://paperswithcode-cyber.com/orgs/meta)

## Code repositories

- [meta\-llama/PurpleLlama](https://github.com/meta-llama/PurpleLlama)

## Paper resources

- [Abstract page](https://arxiv.org/abs/2408.01605v2)
- [PDF](https://arxiv.org/pdf/2408.01605v2.pdf)