Security Data Science at Cloud Scale

Hyperscale cloud computing platforms offer a rich set of services that handle billions of transactions and generate petabyte scale logs. A single service like Azure Active Directory in Microsoft’s cloud handles 450 billion logins per month generating over 10 Petabyte of logs annually. The Azure Security Data Science team is tasked with detecting malicious activities and behaviors in our cloud services by employing data driven approaches to security.

In this talk, we present three classes of problems. First, we show how to detect unusual logins behavior in AAD, a foundational Azure service – which at first pass may look trivial, but at a scale of 5 billion enterprise logins per day, even a small amount of false positive rate can cripple security analysts. We discuss our solution that employs a combination of random walks and Markov chains that drastically reduces false positive rates. Next, we share our approach for combining detections from multiple Azure services in a graphical model, giving us the ability to detect complex multistage attacks in the cloud. Finally, we discuss approaches towards determining user risk score, addressing questions like: how does one model risk? What are the different components that contribute to the riskiness of a user? We answer these questions by sharing our experience in building unsupervised risk score function that integrates various detections that we have.