VulnGPT: An LLM-based Agent for Vulnerability Information Summarization

In modern networks, security depends on system administrators' (sysadmins) ability to efficiently patch known vulnerabilities in software. To do this without disrupting network operations, sysadmins must determine the vulnerability's impact, the likelihood of exploitation, how to patch or deploy other mitigations, and what impact patching will have on operations. Prior work found processing available information to make this decision is a major challenge for sysadmins. One potential solution is to use large-language model-based AI agents to perform the information collection and present sysadmins a vulnerability information summary. This is promising, but introduces a different potential issues inherent to AI agents (e.g., hallucinations) and questions of human-AI interaction.

We perform an initial investigation of this approach's utility by manually assessing the accuracy of vulnerability information from one popular AI agent, ChatGPT, for 50 vulnerabilities. We find ChatGPT is mostly accurate, but introduces some errors. We then introduce VulnGPT, a modular system built around ChatGPT, which focuses the AI agent on websites with relevant information to avoid ChatGPT's inaccuracies, and enables users to incorporate local information. We also discuss how this system can be used to enable future research into sysadmin-AI collaboration during patching.