Lessons Learned from 10k Experiments to Compare Virtual and Physical Testbeds


Jonathan Crussell, Thomas M. Kroeger, David Kavaler, Aaron Brown, and Cynthia Phillips, Sandia National Laboratories

Short Experience Paper


Virtual testbeds are a core component of cyber experimentation as they allow for fast and relatively inexpensive modeling of computer systems. Unlike simulations, virtual testbeds run real software on virtual hardware which allows them to capture unknown or complex behaviors. However, virtualization is known to increase latency and decrease throughput. Could these and other artifacts from virtualization undermine the experiments that we wish to run?

For the past three years, we have attempted to quantify where and how virtual testbeds differ from their physical counterparts to address this concern. While performance differences have been widely studied, we aim to uncover behavioral differences. We have run over 10,000 experiments and processed over half a petabyte of data. Complete details of our methodology and our experimental results from applying that methodology are published in previous work. In this paper, we describe our lessons learned in the process of constructing and instrumenting both physical and virtual testbeds and analyzing the results from each.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

@inproceedings {238252,
author = {Jonathan Crussell and Thomas M. Kroeger and David Kavaler and Aaron Brown and Cynthia Phillips},
title = {Lessons Learned from 10k Experiments to Compare Virtual and Physical Testbeds},
booktitle = {12th USENIX Workshop on Cyber Security Experimentation and Test (CSET 19)},
year = {2019},
address = {Santa Clara, CA},
url = {https://www.usenix.org/conference/cset19/presentation/crussell},
publisher = {USENIX Association},
month = aug