Five Pitfalls for Benchmarking Big Data Systems

Performance is an increasingly important attribute of Big Data systems as focus shifts from batch processing to real-time analysis and to consolidated multi-tenant systems. One of the little-understood challenges in scaling data systems is properly defining and measuring performance. The complexity, diversity, and scale of big data systems make this a difficult task and we frequently encounter haphazard benchmarks that lead to bad technology choices, poor purchasing decisions, and suboptimal cluster operations. This talk draws on performance engineering and field services experience from a leading Big Data vendor. We will talk about the most common performance benchmarking pitfalls and share practical advice on how to avoid them with rigorous metrics and measurement methods.