The New Storage Applications: Lots of Data, New Hardware and Machine Intelligence

We are experiencing today the intersection of multiple trends, each of which changes the storage and data landscape in powerful ways. Machines and applications generate massive quantities of new data whose value degrades over time unless it can be efficiently turned into insight. Hardware innovations, from CPU/DRAM scaling to Flash/Persistent Memory, enable data analytics ranging from classic database queries to machine learning and deep learning. New applications such as Flink, Spark, Beam, etc. have emerged for batch and real-time data processing while in-memory database technologies have also expanded to address these workloads. This talk will describe these trends, the applications and operating system stacks, the problems being solved, and new technical opportunities and challenges.