eSGD: Communication Efficient Distributed Deep Learning on the Edge

Authors: 

Zeyi Tao and Qun Li, College of William and Mary

Abstract: 

Training machine learning model on IoT device is a natural trend due to the growing computation power and the great ability to collect various data of modern IoT de- vice.In this work, we consider an edge based distributed deep learning framework in which many edge devices collaborate to train a model while using an edge server as the parameter server. However, the high network communication cost of synchronizing gradients and parameters between edge devices and cloud is a bottleneck. We propose a new method called edge Stochastic Gradient Descent (eSGD) for scaling up edge training of convolutional neural networks. eSGD is a family of sparse schemes with both convergence and practical performance guarantees. eSGD includes two mechanisms to improve the first order gradient based optimization of stochastic objective functions in edge scenario. First, eSGD determines which gradient coordinates are important and only transmits important gradient coordinates to cloud for synchronizing. This important update can aggressively reduce the communication cost. Second, momentum residual accumulation is designed for tracking out-of-date residual gradient coordinates to avoid low convergence rate caused by sparse updates. Our experiments show that we reach 91.2%, 86.7%, 81.5% accuracy on MNIST data set with gradient drop ratio 50%, 75%, 87.5% respectively.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

BibTeX
@inproceedings {216799,
author = {Zeyi Tao and Qun Li},
title = {eSGD: Communication Efficient Distributed Deep Learning on the Edge},
booktitle = {{USENIX} Workshop on Hot Topics in Edge Computing (HotEdge 18)},
year = {2018},
address = {Boston, MA},
url = {https://www.usenix.org/conference/hotedge18/presentation/tao},
publisher = {{USENIX} Association},
month = jul,
}