Reliable Data Processing with Minimal Toil

Note: Presentation times are in Coordinated Universal Time (UTC).

Thursday, 14 October, 2021 - 16:3017:00

Pieter Coucke, Google; Julia Lee, Slack


Learn about the risks involved with data processing pipelines and how Google and Slack mitigate them.

This talk shares insights into making batch jobs safer and less manual. We have researched and implemented ways to do canarying, automated global rollouts on increasingly larger target populations and different kinds of validations. All these are necessary to remove the manual work involved in updating a batch job globally across millions of users.

Pieter Coucke, Google

Pieter Coucke is a Technical Program Manager at Google SRE Zürich, working within Google Workspace with teams like Gmail, Drive, Calendar, Meet, and Docs.

Julia Lee, Slack

Julia Lee is a Senior Software Engineer in Infrastructure at Slack, where she leads the development of asynchronous compute services.

SREcon21 Open Access Sponsored by Indeed

@conference {276703,
author = {Pieter Coucke and Julia Lee},
title = {Reliable Data Processing with Minimal Toil},
year = {2021},
publisher = {USENIX Association},
month = oct

Presentation Video