You are here
Continuous Pipelines at Google
Dan Dennison, Google, Inc.
This presentation will focus on the real life challenges of managing data processing pipelines of any depth and complexity. I’ll cover the frequency continuum between periodic pipelines that run very infrequently up through continuous pipelines that never stop running, discussing the discontinuities along the axis that can produce significant operational problems. A fresh take on the master-slave model is presented as a better alternative to the periodic pipeline for reliable Big Data scaling.
Dan Dennison has been a Site Reliability Engineer at Google for the past eight years, working on various large pipelines in Ads Quality. The Ads Quality team at Google improves, expands, and supports one of the world’s largest machine learning deployments. Ads Quality also crawls and indexes all advertisements, and enforces policies on new and existing ads.
Open Access Media
USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.