Be Sensitive and Collaborative: Analyzing Impact of Coverage Metrics in Greybox Fuzzing

Authors: 

Jinghan Wang, University of California, Riverside; Yue Duan, Cornell University; Wei Song, Heng Yin, and Chengyu Song, University of California, Riverside

Abstract: 

Coverage-guided greybox fuzzing has become one of the most prevalent techniques for finding software bugs. Coverage metric, which decides how a fuzzer selects new seeds, is an essential parameter of fuzzing and can greatly affect the results. While there are many existing works on the effectiveness of different coverage metrics on software testing, little is known about how different coverage metrics could actually affect the fuzzing results in practice. More importantly, it is unclear whether there exists one coverage metric that is absolutely superior than all the other metrics. In this paper, we report the first systematic study on the impact of different coverage metrics in fuzzing. To this end, we formally define and discuss the concept of sensitivity which can be used to theoretically compare different coverage metrics. We then present several coverage metrics with their variants. We conduct a study on these metrics with the DARPA CGC dataset, the LAVA-M dataset, and a set of real-world applications (a total of 221 binaries). We find that because each fuzzing instance has limited resources (time and computation power), (1) each metric has its unique merit in terms of flipping certain types of branches (thus vulnerability finding) and (2) there is no grand slam coverage metric that defeats all the others. We also explore combining different coverage metrics through cross-seeding and the result is very encouraging: this pure fuzzing based approach is able to crash at least the same numbers of binaries in CGC dataset as a previous approach (Driller) that combines fuzzing and concolic execution. At the same time, our approach uses fewer computing resources.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

BibTeX
@inproceedings {241986,
author = {Jinghan Wang and Yue Duan and Wei Song and Heng Yin and Chengyu Song},
title = {Be Sensitive and Collaborative: Analyzing Impact of Coverage Metrics in Greybox Fuzzing},
booktitle = {22nd International Symposium on Research in Attacks, Intrusions and Defenses ({RAID} 2019)},
year = {2019},
isbn = {978-1-939133-07-6},
address = {Chaoyang District, Beijing},
pages = {1--15},
url = {https://www.usenix.org/conference/raid2019/presentation/wang},
publisher = {{USENIX} Association},
month = sep,
}