Many software vendors conduct or commission penetration testing of their products. In a penetration test security experts identify entry points for attacks in a software product. The audits can be an eye-opener for development teams: they realize that security requires much more attention. However, it is unclear what lasting benefits developers can reap from penetration tests. We report from a one-year study of a penetration test and its aftermath at a major software vendor, and ask how an agile development team managed to incorporate the test findings. Results suggest that penetration tests improve developers' security awareness, but long-lasting change of development practices is hampered if security is not properly reflected in the communicative and collaborative structures of the organization, e.g. by a dedicated stakeholder. Based on our findings we suggest improvements to current penetration test consultancies by addressing communication and organizational factors in software development.