---
title: "AutoPentest: Enhancing Vulnerability Management With Autonomous LLM Agents (arXiv:2505.10321) | Papers With Code Cyber"
description: "A recent area of increasing research is the use of Large Language Models (LLMs) in penetration testing, which promises to reduce costs and thus allow for higher frequency…"
canonical: "https://paperswithcode-cyber.com/paper/2505.10321"
type: "paper"
---

# AutoPentest: Enhancing Vulnerability Management With Autonomous LLM Agents

arXiv:2505\.10321 · 2025-05-15

Authors: Julius Henke

## Abstract

A recent area of increasing research is the use of Large Language Models \(LLMs\) in penetration testing, which promises to reduce costs and thus allow for higher frequency\. We conduct a review of related work, identifying best practices and common evaluation issues\. We then present AutoPentest, an application for performing black\-box penetration tests with a high degree of autonomy\. AutoPentest is based on the LLM GPT\-4o from OpenAI and the LLM agent framework LangChain\. It can perform complex multi\-step tasks, augmented by external tools and knowledge bases\. We conduct a study on three capture\-the\-flag style Hack The Box \(HTB\) machines, comparing our implementation AutoPentest with the baseline approach of manually using the ChatGPT\-4o user interface\. Both approaches are able to complete 15\-25 % of the subtasks on the HTB machines, with AutoPentest slightly outperforming ChatGPT\. We measure a total cost of \\$96\.20 US when using AutoPentest across all experiments, while a one\-month subscription to ChatGPT Plus costs \\$20\. The results show that further implementation efforts and the use of more powerful LLMs released in the future are likely to make this a viable part of vulnerability management\.

## Tasks

- [Penetration testing](https://paperswithcode-cyber.com/tasks/cyberintel-penetration-testing)

## Code repositories

- [JuliusHenke/autopentest](https://github.com/JuliusHenke/autopentest)

## Paper resources

- [Abstract page](https://arxiv.org/abs/2505.10321v1)
- [PDF](https://arxiv.org/pdf/2505.10321v1.pdf)