NEMESYS: Network Message Syntax Reverse Engineering by Analysis of the Intrinsic Structure of Individual Messages

Protocol reverse engineering based on traffic traces allows to analyze observable network messages. Thereby, message formats of unknown protocols can be inferred. We present a novel method to infer structure from network messages of binary protocols. The method derives field boundaries from the distribution of value changes throughout individual messages. None of many previous approaches exploits features of structure which are contained within each single message. Our method exploits this intrinsic structure instead of comparing multiple messages with each other. We implement our approach in the tool NEMESYS: NEtwork Message SYntax analysiS. Additionally, we introduce the Format Match Score: the first quantitative measure of the quality of a message format inference. We apply the Format Match Score to NEMESYS and a previous approach and compare the results to mutually validate our new format inference method and the measure of its quality.