Gaurav Nanda and Vincent Cheng, Databricks
Kubernetes relies on kube-proxy and DNS for simple Layer 4 load balancing, which works for short-lived HTTP traffic but fails for persistent connections and high-throughput gRPC workloads. With thousands of requests multiplexed over a single TCP connection, clusters often see uneven load, pod hot-spotting, and rising tail latency.
This talk presents a client-side, control-plane-driven approach that removes kube-proxy and DNS from the data path. A lightweight control plane tracks Service and EndpointSlice updates, while client libraries receive live endpoint changes through xDS and make per-request routing decisions at Layer 7. We show how strategies like Power of Two Choices and zone-affinity routing improve load balance, stabilize tail latency, and reduce resource waste in production.
SREs and platform engineers will learn why default Kubernetes routing breaks down, how to design intelligent client-side load balancing, and what operational challenges emerge when deploying these systems at scale.
Gaurav Nanda leads the Application Traffic and Networking Platform Infrastructure group at Databricks, where he focuses on multi cloud connectivity, intelligent load balancing, and overload protection for large scale Data and AI systems. He brings more than fifteen years of experience and previously held engineering leadership roles at Google and Harness.
Vincent is a software engineer on the Application Traffic team at Databricks. Much of his present day work involves ensuring that internal services can seamlessly reach out to each other and distribute network traffic efficiently. Prior to Databricks, he worked on large scale configuration distribution for Google Cloud.
author = {Gaurav Nanda and Vincent Cheng},
title = {Intelligent Load Balancing in Kubernetes},
year = {2026},
address = {Seattle, WA},
publisher = {USENIX Association},
month = mar
}
