A Comprehensive View of a Large-Scale Deep Learning Offline Platform

Lukas Majercak, Shansi Ren, Jack Zhang, and Junhua Wang, Microsoft


In the past several years, based on the Apache OSS technologies, we have built a scalable and efficient ML/DL inference and training platform called MagnetTar (MT). Our MT platform has been running on top of tens of thousands of Windows servers and has been widely used across Bing, Bing Ads, and News. MT has empowered many critical large-scale offline batch and streaming scenarios.

Lukas Majercak, Microsoft

Lukas is a developer in the Bing MT team. Lukas has been a key contributor in many areas in MT, such as HDFS, Spark, and Deep-Learning Offline Processing, etc.

Shansi Ren, Microsoft

Shansi is an engineering manager in the Bing MT team. Prior to MT, he had worked on Index Update and Serving technologies for many years in Bing.

@conference {232939,
author = {Lukas Majercak and Shansi Ren and Jack Zhang and Junhua Wang},
title = {A Comprehensive View of a Large-Scale Deep Learning Offline Platform},
year = {2019},
address = {Santa Clara, CA},
publisher = {{USENIX} Association},
month = may,