KubeCon + CloudNativeCon North America 2019

Flyte is the backbone for large-scale Machine Learning and Data Processing (ETL) pipelines at Lyft. It is used across business critical applications ranging from ETA, Pricing, Mapping, Autonomous, etc. At its core is a Kubernetes native workflow engine that executes 10M+ containers per month as part of thousands of workflows. The talk will focus on,
- Architecture of Flyte and its specification language to orchestrate compute and manage data flow across disparate systems like Spark, Flink, Tensorflow, Hive, etc.
- Deploying highly scalable and fault tolerant Kubernetes Operators
- Learnings from operating Flyte across multiple Kubernetes clusters and using other CNCF technologies like gRPC, Envoy, FluentD, Kustomize and Prometheus.
- Use-cases where Flyte can be leveraged
The talk will conclude with a demo of a machine learning pipeline built using the open source version of Flyte.