How to Make Releases Safer in Baidu

Changes/updates are a major source of service faults. In Baidu, around 54% of the faults are introduced by changes. As a result, progressive rollout becomes imperative to improve service stability. Progressive rollout divides the deployment process into several stages. Each stage only deploys the change on a subset of the instances. Checkings are applied between consecutive stages to detect faults. If a fault is detected, the deployment is terminated and rolled back.

Intuitively, we can build a rollout system that enables development engineers to specify checking rules in each stage. Surprisingly, however, the Devs are not good at this, although they are the creators of the modules. Therefore, the reliability engineers are forced to add rules on stability indicators. But this leads to numerous false alarms, stalling the release procedure frequently. As a result, we turn to machine learning based methods. In order to obtain satisfying results, the algorithm must be able to learn the “normal” changes of each indicators, and quantitatively measure current changes to decide whether there are faults or not.

In this talk, we will present several real cases to demonstrate the dilemma we confront in rollout checking, and how the machine learning algorithm works.