In order for the field of computer security to progress, we need to know the true value of different security technologies and understand how they work in practice. It is important to evaluate their effectiveness as they are being used in an ecologically valid environment. To this end, we postulate that security products could be evaluated by conducting computer security clinical trials. To show the feasibility of such approach, we did a 4-month proof of concept study with 50 users, that aimed to evaluate an anti-malware product. In this paper, we present the study we performed and provide lessons learned and recommendations on the challenges, limitations and considerations of conducting computer security clinical trials.