---
title: "CRAKEN: Cybersecurity LLM Agent with Knowledge-Based Execution (arXiv:2505.17107) | Papers With Code Cyber"
description: "Large Language Model (LLM) agents can automate cybersecurity tasks and can adapt to the evolving cybersecurity landscape without re-engineering. While LLM agents have…"
canonical: "https://paperswithcode-cyber.com/paper/2505.17107"
type: "paper"
---

# CRAKEN: Cybersecurity LLM Agent with Knowledge\-Based Execution

arXiv:2505\.17107 · 2025-05-21

Authors: Minghao Shao, Haoran Xi, Nanda Rani, Meet Udeshi, Venkata Sai Charan Putrevu, Kimberly Milner, Brendan Dolan\-Gavitt, Sandeep Kumar Shukla, Prashanth Krishnamurthy, Farshad Khorrami, Ramesh Karri, Muhammad Shafique

## Abstract

Large Language Model \(LLM\) agents can automate cybersecurity tasks and can adapt to the evolving cybersecurity landscape without re\-engineering\. While LLM agents have demonstrated cybersecurity capabilities on Capture\-The\-Flag \(CTF\) competitions, they have two key limitations: accessing latest cybersecurity expertise beyond training data, and integrating new knowledge into complex task planning\. Knowledge\-based approaches that incorporate technical understanding into the task\-solving automation can tackle these limitations\. We present CRAKEN, a knowledge\-based LLM agent framework that improves cybersecurity capability through three core mechanisms: contextual decomposition of task\-critical information, iterative self\-reflected knowledge retrieval, and knowledge\-hint injection that transforms insights into adaptive attack strategies\. Comprehensive evaluations with different configurations show CRAKEN's effectiveness in multi\-stage vulnerability detection and exploitation compared to previous approaches\. Our extensible architecture establishes new methodologies for embedding new security knowledge into LLM\-driven cybersecurity agentic systems\. With a knowledge database of CTF writeups, CRAKEN obtained an accuracy of 22% on NYU CTF Bench, outperforming prior works by 3% and achieving state\-of\-the\-art results\. On evaluation of MITRE ATT&CK techniques, CRAKEN solves 25\-30% more techniques than prior work, demonstrating improved cybersecurity capabilities via knowledge\-based execution\. We make our framework open source to public https://github\.com/NYU\-LLM\-CTF/nyuctf\_agents\_craken\.

## Tasks

- [Penetration testing](https://paperswithcode-cyber.com/tasks/cyberintel-penetration-testing)

## Code repositories

- [NYU\-LLM\-CTF/nyuctf\_agents\_craken](https://github.com/NYU-LLM-CTF/nyuctf_agents_craken)

## Paper resources

- [Abstract page](https://arxiv.org/abs/2505.17107v1)
- [PDF](https://arxiv.org/pdf/2505.17107v1.pdf)