What's Changing in Big Data?

Big data analytics became a hot research topic nearly ten years ago, but since that time, a lot of things have changed. On the hardware side, trends such as the slowdown of processing with respect to I/O are starting to affect the design of big data systems. On the application side, big data systems are increasingly being used by non-programmers and require similar forms of interaction to "small data" analysis tools. Finally, big data systems are increasingly provided "as a service" on cloud infrastructure. I'll talk about these changes from the perspective of the Apache Spark project and from my experience at a company offering a cloud service for big data (Databricks).

Matei Zaharia is an assistant professor of computer science at MIT as well as CTO of Databricks, the company commercializing Apache Spark. He is broadly interested in computer systems, data centers and data management. He started the Spark project while he was a PhD student at UC Berkeley, and he has also contributed to other open source cluster computing projects such as Apache Mesos and Apache Hadoop. Matei received the 2014 ACM Doctoral Dissertation Award for his graduate work.