Loading…
Wednesday, November 20 • 10:55am - 11:30am
Running Large-Scale Stateful Workloads On Kubernetes at Lyft - Surinder Singh & Anmol Khurana, Lyft

Sign up or log in to save this to your schedule and see who's attending!

Feedback form is now closed.
Along with core services, K8s at Lyft also forms the base to run a large variety of data processing stateful data processing jobs which includes Spark, Flink and other jobs via various ML and Data processing pipelines.

At Lyft, K8s has become the driver for the majority of our data processing needs running 10s of thousands of concurrent jobs. Operating the platform at this scale presents an unique set of challenges which get more complex with highly variable load pattern.

In this talk, the speakers will share their journey through some of these challenges and learnings.
- Potential pitfalls of running stateful jobs on K8s.
- Knobs/tweaks to optimize K8s for stateful jobs.
- Running k8s in a cloud environment.
- Building a fault-tolerant self-healing system with multiple K8s clusters underneath.

Talk will also focus on optimizations done to support the widely used workloads at Lyft.

Speakers
avatar for Surinder Singh

Surinder Singh

Software Engineer, Lyft
Surinder Singh is a software engineer at Lyft in Seattle. He led execution plane for Flyte, Lyft’s open-source Machine learning and Data processing pipelines platform. Before Lyft, Surinder was at Microsoft where he worked on Azure Storage and SQL Server Query Optimizer.
AK

Anmol Khurana

Software Engineer, Lyft
Anmol Khurana is a software engineer at Lyft. He is part of Data Platform team responsible for leading effort on Containerized Spark on K8s. Before Lyft, Anmol was at Amazon for 5+ years mostly with AWS Elastic Block Store team.



Wednesday November 20, 2019 10:55am - 11:30am
Ballroom Sec 20AB - San Diego Convention Center Upper Level