Pentest assistido por inteligência artificial: um estudo comparativo entre execução manual, LLMs e agente autônomo

Abstract

Penetration testing (pentest) is a well-established security assessment practice, traditionally conducted manually and dependent on the analyst’s expertise. The advance of large language models and autonomous agents raises the question of whether these tools can support or surpass manual execution, yet the literature lacks structured comparative studies evaluating different levels of automation under the same conditions. This dissertation evaluates the applicability of Artificial Intelligence in penetration testing through a systematic literature mapping and practical experiments in a controlled environment. The mapping, conducted following Kitchenham and Charters (2007), identified five primary studies published between 2019 and 2024. The experiments compared three scenarios against Metasploitable 2: manual execution, LLM-assisted execution via command-line interface using Claude Sonnet 4.6 and Gemini 2.5 Pro, and autonomous agent execution via Claude Code CLI. Five metrics were systematically collected: total execution time, number of actions, PTES depth, vector coverage, and action precision. All scenarios achieved full root compromise and 100% vector coverage. Differences appeared in precision: the Claude-assisted scenario reached 85%, below the manual baseline of 96.7%, due to payload suggestions incompatible with the ARM64 architecture of the attacking machine; Gemini reached 96.5%; the autonomous agent exhibited non-deterministic behavior across runs and an average precision of approximately 78%, yielding the lowest degree of reproducibility among the evaluated scenarios. The hypothesis that AI would outperform the human analyst in at least two metrics was not consistently confirmed. The results indicate that general-purpose language models lack awareness of specific technical execution contexts, and that LLM-based autonomous agents still exhibit variability inconsistent with reproducibility, a fundamental property of structured penetration testing methodologies.