Auditing Data Privacy for Machine Learning

Large machine learning models (e.g., deep language models) memorize a significant amount of information about the individual data records in their training set. Recent inference attacks against machine learning algorithms demonstrate how an adversary can extract sensitive information about a model’s training data, by having access to its parameters or predictions. Specifically, these algorithms reflect the identification risk of models by detecting the presence of data records in the training set of a model (hence called membership inference attacks). These attacks can measure the information leakage of models about the data in their training set, thus can be used as a measure to audit privacy risks of machine learning algorithms. Based on the results of these attacks on many real world systems and datasets (e.g., Google and Amazon ML as a service platforms, federated learning algorithms, and models trained on sensitive datasets such as text data, medical, location, purchase history, image data, etc), we conclude that large models pose a significant risk to data privacy of individuals, and need to be considered as some type of personal data. Thus, we need carefully designed methodologies and tools to audit data privacy risk in machine learning in a wide range of applications. Guidances released by the European Commission and the White House call for protection of personal data during all the phases of deploying AI systems and build systems that are resistant to attacks. Recent reports published by the Information Commissioner’s Office (ICO) for auditing AI and the National Institute of Standards and Technology (NIST) for securing applications of Artificial Intelligence also highlight the privacy risk to data from machine learning models. And they specifically mention membership inference as a confidentiality violation and potential threat to the training data from models. It is recommended in the auditing framework by ICO for organizations to identify these threats and take measures to minimize the risk. As the ICO’s investigation teams will be using this framework to assess the compliance with data protection laws, organizations must account for and estimate the privacy risks to data through models. To this end, we have developed an open source tool, named ML Privacy Meter, based on membership inference algorithms, and also tech companies are using similar algorithms to analyze privacy risk in machine learning algorithms. For example, ML privacy meter and similar tools can help in data protection impact assessment (DPIA) by providing a quantitative assessment of privacy risk of a machine learning model. The tool can generate extensive privacy reports about the aggregate-level and individual-level risk with respect to training data records. It can estimate the amount of information that is revealed through the predictions of a model (when deployed) or its parameters (when shared). Hence, when providing query access to the model or revealing the entire model, the tool can be used to assess the potential threats to training data.

In this talk, I will talk about what exactly privacy risk is and what it is not, the difference between privacy and confidentiality (which can be easily confused), the reasons models are vulnerable to inference attacks, the methodology for quantifying privacy risk in machine learning, and examples of how ML privacy meter and similar tools can enable detailed auditing of ML systems. I will show the very fundamental and intuitive relation of the auditing mechanisms and defense mechanisms for privacy (e.g., differential privacy).

It is very important to ML engineers, policymakers, and researchers to be aware of the risks, their implications, and the methodology for auditing the privacy risk for different types of machine learning algorithms. This can pave the way for privacy by design for machine learning.