Clear, Actionable and Confidence-Inspiring Recommendations? Comparative Study of AI-Generated and Human-Written PT Reports

Penetration testing reports typically include remediation recommendations to guide organizations in addressing identified vulnerabilities. The usability of such recommendations influences whether the reports are effectively understood and acted upon. Yet, the usability implications of different recommendation formulations, including AI-generated content, remain underexplored. This paper presents a mixed methods study examining how participants perceive different versions of PT recommendations. In our study, total of 598 IT professionals and cybersecurity students evaluated five recommendation variants for each of two vulnerability descriptions. These variants included versions written by an experienced penetration tester or generated by AI, and differed in the level of detail provided, as well as in the inclusion or exclusion of an explicit responsibility role tag. Study participants evaluated each recommendation using scaled ratings for three aspects: clarity, actionability, and confidence in applying the guidance. They also selected their preferred recommendation in pairwise comparisons and provided reasoning.

Results reveal some significant differences in perceived usability across the recommendation variants. Recommendations with clear steps and an organized structure were rated higher for clarity, actionability, and confidence. Presenting guidance as a sequence of actionable steps helps users quickly understand what to do, reduces ambiguity, and decreases the time needed to interpret and implement the recommendation. These results suggest that both human and AI authors of PT reports should prioritize structured presentation to make security guidance easier to understand and act on.