InStream: Large Scale Distribution using BitTorrent, Python, Salt, and Kafka

Wednesday, 2017, August 30 - 14:3015:00

Harsh Sharma, LinkedIn

Abstract: 

Deploying application/services to all the servers across every datacenter can be painful for any company with a big infrastructure, including LinkedIn.

Our deployment model had some centralized pieces which became bottlenecks at scale. This talk will describe how we built a service in Python, based on Saltstack and Kafka, which can deploy any service to all servers asynchronously with a P2P distribution model, rate limiting and fast rollbacks.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

Presentation Audio

BibTeX
@conference {205484,
author = {Harsh Sharma},
title = {InStream: Large Scale Distribution using BitTorrent, Python, Salt, and Kafka},
year = {2017},
address = {Dublin},
publisher = {{USENIX} Association},
}