MPP: Model Performance Predictor

Authors: 

Sindhu Ghanta, Sriram Subramanian, Lior Khermosh, Harshil Shah, Yakov Goldberg, Swaminathan Sundararaman, and Drew Roselli, ParallelM; Nisha Talagala, Pyxeda AI

Abstract: 

Operations is a key challenge in the domain of machine learning pipeline deployments involving monitoring and management of real-time prediction quality. Typically, metrics like accuracy, RMSE etc., are used to track the performance of models in deployment. However, these metrics cannot be calculated in production due to the absence of labels. We propose using an ML algorithm - Model Performance Predictor, to track the performance of the models in deployment. We argue that an ensemble of such metrics can be used to create a score representing the prediction quality in production. This in turn facilitates formulation and customization of ML alerts, that can be escalated by an operations team to the data science team. Such a score automates monitoring and enables ML deployments at scale.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

BibTeX
@inproceedings {232987,
author = {Sindhu Ghanta and Sriram Subramanian and Lior Khermosh and Harshil Shah and Yakov Goldberg and Swaminathan Sundararaman and Drew Roselli and Nisha Talagala},
title = {{MPP}: Model Performance Predictor},
booktitle = {2019 {USENIX} Conference on Operational Machine Learning (OpML 19)},
year = {2019},
isbn = {978-1-939133-00-7},
address = {Santa Clara, CA},
pages = {23--25},
url = {https://www.usenix.org/conference/opml19/presentation/ghanta},
publisher = {{USENIX} Association},
month = may,
}