Addressing Some of Challenges When Optimizing Long-to-Train-Models

Tobias Andreasen, SigOpt

Abstract: 

As machine learning models become more complex and require longer training cycles, optimizing and maximizing performance can sometimes be seen as an intractable problem - this tends to leave a lot of performance unrealized.

The challenge oftentime becomes that most common methods for hyperparameter optimization are either sample efficient or they are able to efficiently parallelize. This either leads to a choice between a very long optimization process with good performance, or a very short but efficient optimization process with suboptimal performance.

Further, another challenge becomes justifying the cost of optimizing these oftentime long-to-train-models, because in most situations this has to be done on a per model basis with non of information gained being leverage in the future.

This talk outlines ways in which these challenges can be addressed, when thinking about bringing optimal performing models into production.

Tobias Andreasen, SigOpt

Tobias Andreasen is a Machine Learning Specialist at the San Francisco based startup, SigOpt. A company with the vision to “Accelerate and amplify the impact of modelers everywhere” through the development of software for optimization and experimentation. On a day-to-day basis Tobias is working with a large set of companies, on how to come up with the best approach to optimizing for things like their machine learning models in order to meet their business constraints and requirements.

Tobias holds a master’s and bachelor’s degree from the Technical University of Denmark within the field of applied mathematics.

OpML '20 Open Access Sponsored by NetApp

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

BibTeX
@conference {256646,
author = {Tobias Andreasen},
title = {Addressing Some of Challenges When Optimizing Long-to-Train-Models},
year = {2020},
publisher = {{USENIX} Association},
month = jul,
}

Presentation Video 
Teaser
Full