Alexey Skorikov, Google
Growing complexity of data pipelines and organizations poses more challenges to Data Reliability. Risk of data incidents multiplies with each hop downstream. Teams see decrease in operational readiness to deal with specific classes of outages: data staleness, corruption and loss, while costs of incident resolution and revenue impact of outages can grow non-linearly. Without understanding the full data dependency graph, it is hard to measure completeness of monitoring, leading to gaps. For any sizable organization, figuring out data dependencies manually is a non-starter. We will talk about:
- Understanding critical business data flows, upstream & downstream dependencies;
- Holistic data monitoring at scale to eliminate unnoticed data staleness, which potentially; can lead to accumulated negative business impact;
- One of the approaches Google took in this direction and lessons learned;
- How you can practically instrument this approach in your organization.
Alexey Skorikov, Google
Alexey Skorikov is a Technical Program Manager for Google Ads, specializing in Site Reliability Engineering (SRE), and working on programs for data reliability and data pipelines efficiency. Prior to Google, he helped tech companies in the US and EU to manage software programs, cross-functional engineering teams, and organizational changes.
Alexey has over 15 years of hands-on software engineering experience, having specialized in SaaS, network services, business processes automation, and data processing, across several industries such as GIS, AdTech, ISP, E-commerce, Oil & Gas.
As a speaker, Alexey delivered four distinct talks at professional conferences and five workshops for software engineers and managers.
SREcon21 Open Access Sponsored by Indeed
author = {Alexey Skorikov},
title = {Horizontal Data Freshness Monitoring in Complex Pipelines},
year = {2021},
publisher = {USENIX Association},
month = oct
}
