SREcon25 Europe/Middle East/Africa Program

SREs are under pressure to "just add AI" without increasing risk. But where do you start, and how do you avoid hype-driven dead ends? This talk shares lessons from 18 months of applying LLMs in a real, high-stakes environment: powering live gaming, simulation, and ML inference. We’ll walk through use-cases that worked, the pitfalls we encountered, and how we addressed challenges like hallucinations, alert fatigue, and on-prem LLM deployment. The session includes hands-on demos of playbook indexing, live signal integration, and role-specific outputs, all built into a practical AI stack that keeps your data within your firewall. You’ll leave with honest takeaways, working examples, and clear steps for responsibly integrating AI into your operations.

This talk offers a deep dive into managing Google's planet-scale Build & CI stack from an SRE perspective. We'll explore the complexities of supporting over 100,000 monthly users and massive workloads, maintaining a 98% cache hit rate across diverse computing environments. Discover resource management strategies for doubling year-over-year growth, the application of the Pareto principle, and critical caching layers (local, remote, P2P). We'll cover output storage managing hundreds of petabytes daily through deduplication and compression, and trace a user's build journey from desktop to production. Gain insights into optimizing the build stack and maintaining a 24/7 service with a small SRE team through strategic planning, monitoring, and collaboration. Finally, we'll discuss availability risks, standardization, and the pros and cons of a monolithic Build & CI stack.

Scaling a generative AI platform is no fairy tale. Instead, it’s an epic battle through dungeons spanning training workloads, inference services, and infrastructure. GenAI has introduced a whole new level of complexity for infrastructure, bringing heavier resource demands, new requirements for the scaling of token-based patterns, and questions about how to monitor and manage it all.

Building GenAI systems is hard, but keeping it reliable is even harder.

In this talk, we’ll recount our journey taming the complexity of multi-cluster AI platforms using actionable SLOs as our compass. Whether you’re building your first AI platform or defending the reliability of a cluster, you’ll complete your quest equipped with practical, open source-friendly strategies to help make your systems observable, debuggable, and resilient.

In complex microservices architectures, a single frontend request can traverse dozens of service nodes. Under high concurrency, this creates tightly coupled scheduling chains where minor performance jitters—when combined with retries and queuing—can trigger avalanche failures. These occur when slow or failed services cause upstream retries and thread pool exhaustion, leading to blocked queues and cascading system-wide outages. Even after the initial fault is resolved, stuck requests may still clog the system, prolonging downtime.

This work presents a systematic approach to prevent and mitigate such failures. Key strategies include: fine-tuning timeout and retry settings, introducing full-link retry budgeting to reduce retry storms, and applying adaptive degradation and queue-shedding techniques during overloads. By breaking the positive feedback loop of retries and queuing, systems can recover faster and maintain availability—even under fault conditions.

Running AI inference on the geographically distributed network presents a unique set of challenges that traditional cloud-centric SRE practices often fail to address. Unlike centralized data centers, edge deployments involve geographically dispersed processing units, vast model catalogs, and a complex interplay of resource constraints and network variability. A critical, yet often elusive, objective in this domain is achieving high processing unit utilization, which directly impacts both the operational cost (CapEx efficiency) and service quality (latency trade-offs). Underutilized models are a significant financial burden, while over-utilization can degrade user experience.

SREs working on build and engineers working in DevOps are likely familiar with container orchestration and build systems using Docker, but may not necessarily be familiar with some of the latest developments in build tooling. Docker Bake is a powerful new toolchain embedded into the Docker buildkit daemon that allows for decomposing Docker images into reusable steps and parallel building of multiple Docker images from the same sources at once. This allows for faster builds, with better provenance. Learn how Honeycomb slashed our build times in half using Docker Bake.

Service Meshes are extremely trendy right now, and the benefits are many. Traffic management, flow control, multi-cluster load balancing, mTLS are all advertised benefits. But of course all these are benefits of having a service, and unless your company is brand new, nobody starts with one.

Pagerduty featured around 500 individual services needed to be migrated. What's worse, we had a very lean team to do it with. We'd use the "away team" model, where we made all the initial code changes required, then asked other teams nicely to review those changes.

It seemed like a nearly impossible task, but a year later we're looking at 95% completion and making good progress on the long tail. Along the way we've learned a lot about how to make a migration successful, what's good in a service mesh, and how to keep a team from burning out on an extremely monotonous task.

Leader election is a popular design pattern for distributed systems managing critical state. But despite its simple and innocent appearance, hidden dangers lurk. A reliable leader-elected service requires more than just a proven consensus implementation and a superficial strategy for handling the lower availability that comes with strong consistency.

This talk will present the theory and practice of fencing, illustrated by a serious outage of a Google service which we thought was doing everything right. It will also discuss some challenges with the operation of leader elected services, and some alternatives to leader election.

MLOps isn’t just about getting models into production. It’s about making them reliable, traceable, and easy to maintain. In this workshop, we’ll explore the core principles of MLOps and put them into practice, focusing on traceability and reproducibility in real-world ML systems.

While the principles are tool-agnostic, we’ll demonstrate them using MLflow and Databricks: tracking models, deploying pipelines, and showing how these practices help you move fast without breaking production.

As a public cloud company, we needed an internal reliable environment to safely test changes before shipping to Production. Supporting our infrastructure requires validating an increasingly widening array of components and services. This required our SRE team to build and maintain a full-featured replica of Production from scratch. This talk tells the story of how we brought "Testcloud" online. We’ll discuss a technical overview of the challenges we faced building our "Testcloud" including how we tackled the unique operational challenges that come with building isolated ecosystems. These include how we evaluated and adapted existing industry examples, challenges in achieving parity with production, partitioning production from lower environments, and managing circular dependencies. The target audience is anyone who finds themselves having to test software code or physical components while managing customer expectations.

Training engineers on the skills to be effective incident commanders (communication, coordination, delegation, etc.) can be a challenge—we tend to get bogged down in whether a scenario is "realistic" enough—and then derailed by debugging, instead of focusing on incident command itself. What if we solved for that by removing engineering from practice scenarios entirely? This talk presents a real, in-use training and practical exercise for incident command skills doing just that, by using a made-up, lighthearted scenario (with Pokemon!).

Every growing company hits the same wall: skyrocketing costs from metrics and logs. Riot Games was no exception: ingesting over a petabyte a month, we had to rethink everything.

In this talk, we’ll share how we tackled our ballooning observability bill. Not just with tools, but by reshaping engineering culture to make cost a shared responsibility. From clever process tweaks to accountability frameworks, we’ll show you what actually moved the needle.

Whether you're an IC drowning in dashboards or a leader staring down a budget cliff, this session will give you the insights (and battle scars) to take control of your telemetry spend.

In a statement unsurprising to just about everybody, achieving reliability isn’t exactly straightforward. What would you do if your mission is to enable the teams in your organisation to progress towards it? You’re outnumbered and competing for service owners' finite attention against alternate incentives.

This is an honest reflection on techniques we relied on to enable teams to understand and make steps towards improving their reliability posture. We’ll look at some of the major activities we undertook along our journey as the sociotechnical system grew around us. Scaling our SRE-in-the-loop operational readiness assessments whilst trying to mitigate common reliability issues with platform primitives had us stretched thin. We invested in building out a reliability data platform for continuous attestation to help us scale our efforts. We'll also discuss our service review programme that involved collaborating with teams to surface architectural and operational readiness issues that they were experiencing.

Not every learning moment comes from a declared incident. Across many years in SRE and incident response, I’ve seen that a significant portion — often between 30% and 60% — of potential incidents never make it into formal tracking. While this reduces the immediate load on response teams, it also means we miss critical opportunities to learn.

This talk explores the "Un-Incident" — those ambiguous, gray-zone events that don’t meet formal criteria but still reveal gaps in systems, processes, or assumptions. You’ll learn how to recognize these moments, reduce friction in triage and reviews, and shift from debating classification to extracting insight.

With real-world examples and practical strategies I’ve learned over the years, this session offers a fresh lens on incident response — one that values learning over labels.

Driving platform adoption is hard, especially when feature teams are laser-focused on delivery metrics and SRE teams are overstretched. In this talk, we introduce a hybrid model that blends platform engineering and SRE to tackle both challenges: the Platform SRE.

Platform SREs focus on building self-service reliability features directly into the platform, so new services are production-ready from day one.

But for legacy or business-critical systems, they take a different approach: embedding temporarily into feature teams to lead focused reliability initiatives like observability migrations, alerting, or SLO alignment.

This time-boxed, tactical engagement accelerates platform adoption while reducing the operational burden on central SRE teams.

Attendees will hear real-world insights from implementing this model inside a mid-size engineering org, along with lessons on team design, cultural prerequisites, and common pitfalls to avoid.

Ideal for platform engineers, SREs, and engineering leaders driving operational excellence at scale.

Our decentralised operational model effectively supported our system for years, but we recognised it might not scale with increasing complexities and reliability expectations. To explore a solution, we initiated a deliberate trial embedding an SRE within a key trading systems engineering team. Working directly alongside the engineers enabled us to drive operational accountability, align closely with organisational OKRs, and build the trust necessary for meaningful reliability improvements from within.

We navigated unique constraints; traditional tools like error budgets and gradual SLO-based alerting were incompatible with finances zero tolerance for delays. Instead, we reshaped alerting around practical telemetry and revitalised incident retrospectives, enhancing their effectiveness and drawing out actionable insights. Over 12 months, this approach rebuilt trust, significantly reduced incidents and required no additional headcount.

This session shares practical insights for embedding SRE in your organisation, demonstrating how to adapt standard practices, deliver immediate value, and foster a sustainable reliability culture.

Engineering teams constantly balance deployment velocity with system reliability. This challenge is particularly relevant for critical infrastructure like monitoring platforms that need to be more reliable than the systems they observe.

This talk demonstrates how to build automated quality gates that make deployment decisions based on production monitoring data. I'll show a practical implementation using open-source tools including GitHub Actions, ArgoCD, TestContainers, Playwright, and OpenTelemetry that validates deployments before they reach users.

You'll see the actual pipeline in action, including:

• How we correlate deployment events with error rates, latency, and business metrics using OpenTelemetry traces, logs, and metrics
• Quality gate criteria that catch regressions across data pipelines and application services
• Open-source tooling integration that teams can adapt to their environments

This approach is particularly useful for data-heavy and AI workloads where traditional health checks provide limited insight. Using MLFlow for experiment tracking and model management, we've also implemented quality gates that validate model accuracy, detect drift, and verify inference performance before promoting AI services to production.

Key takeaways include practical quality gate patterns you can implement with existing tools, the specific metrics that indicate deployment success, and lessons learned from operating this system in production. Whether you're deploying traditional applications or AI workloads, you'll gain concrete strategies to improve deployment confidence while maintaining development velocity.

A single switch failure in a cloud provider's network can impact millions of users. So why is it still so difficult to trace the network path of a single customer? While large-scale network monitoring is a solved problem, observing individual tenant flows is not. This talk dives into the complexities of the cloud backbone and introduces a powerful tool for end-to-end flow tracking. You'll learn how this approach helps us rapidly troubleshoot issues and resolve failures that are nearly invisible to traditional, large-scale monitoring systems.

Mollie undertook a major infrastructure initiative to expand from a single-region setup to a multi-region architecture on Google Cloud, aiming to enhance disaster recovery and platform resilience. With rapid growth, high transaction volumes, and a commitment to uptime, the shift was driven by the need for better fault tolerance and business continuity. The project involved significant technical preparation, including updating application code for multi-region awareness, upgrading edge load balancers, re-architecting database failover strategies, and refining disaster recovery runbooks. Execution was carefully planned, following incident response best practices and rigorous testing across environments. The initiative represented a critical step in Mollie’s evolution toward a more resilient, scalable cloud infrastructure—one capable of maintaining performance and availability even in the face of regional disruptions. This case offers valuable insights into the complexities of cloud expansion and the importance of preparation, iteration, and operational discipline when building for resilience at scale.

SREs know about some of the flaws and vulnerabilities in their systems. They might also have intuition on where to look for additional issues–"known unknowns." But what about the "unknown unknowns"–outages waiting to happen that nobody is even looking for? With the vast complexity of modern software systems, this dark space of unknowns can be huge. And, what's worse, most of the outages in this space happen due to complex interactions between various parts of the system, even when everything is working according to specification, i.e. no implementation bugs.

What if we had a way to shine a light into the unknown unknowns? What if we could understand our systems enough to be able to methodically explore these complex interactions and build a comprehensive list of possible outage scenarios? In STPA, we model systems based on control-feedback loops, creating a hierarchical control structure, or HCS. In this session, we'll use a real Google system to show how an HCS can help you gain a new perspective and understanding of your system. We'll note similarities to common patterns in software design so you can start thinking about similar vulnerabilities in your own systems.

This session will be an interactive workshop. Attendees should plan to actively participate in the small group exercises in order to get the most benefit from the session.

In the world of financial systems, data is everything, but not all data is created equal and not all systems are built to handle its volatility. At Bloomberg Engineering, managing market data is a uniquely complex challenge. This plenary will dive into how as Reliability Engineers we build, monitor and evolve our systems to ensure high quality data delivery amidst the chaos of global markets.

We’ll ask the uncomfortable question: what if we didn’t catch it? Through real-world scenarios, we’ll highlight the cost of bad data: from regulatory risk, wasted engineering hours and eroded customer trust.

From the daily issues like file delays, malformed data, and repeated deliveries to the abnormal market occurrences that cause excess load and bespoke handling; there are many tools and techniques familiar to Reliability Engineers that can contribute to the delivery of quality data.

Join us to discover how Reliability Engineers at Bloomberg handle these challenges.

As Data & AI systems become foundational to modern software, the pipelines that power them deserve the same engineering rigor as production services. Yet many software engineers still treat data pipelines as someone else’s responsibility. We’ll discuss why end-to-end lineage is essential for observability, debugging, and trust in Data & AI workflows, how it can allow software engineers to maintain the health of data pipelines and how this aligns with principles familiar to the SRE community. The talk will introduce the OpenLineage standard (a project under LF AI & Data), explain how it compares and complements OpenTelemetry, and present current integrations with Airflow, Spark, dbt, Flink, and more. Presented by a Technical Steering Committee (TSC) member of OpenLineage, this session is both a practical introduction and a call to action: to treat Data & AI pipelines as first-class citizens in reliable, scalable systems.