KubeCon + CloudNativeCon North America 2019

PromQL, the Prometheus Query Language, is a concise, powerful and increasingly popular language for querying time series data. But PromQL queries can take a long time when they have to consider >100k series and months of data. Even with Prometheus’ compression, a 90 day query over 200k series can touch ~100GB of data.

In this talk we will present a series of techniques employed by Cortex (a CNCF project for clustered Prometheus) for accelerating PromQL queries -- namely query results caching, time slice parallelisation, aggregation sharding and automatic recoding rule substitutions.

But there’s more: we will show how you can use this technology to get these improvements with Thanos and Prometheus. Finally, we will cover optimisations to the PromQL engine by the Cortex team, and how these have already been merged upstream to benefit the whole community.

Did you know that by default, your applications running in Kubernetes can open raw network sockets? This talk demonstrates how, in the right circumstances, the CAP_NET_RAW capability that allows this can be abused by a compromised application.

* ARP spoofing: pretending to represent the wrong IP address
* If the app can ARP spoof the IP address of the DNS service, this potentially lets it spoof DNS addresses: pretending to represent the wrong domain name

Sounds bad, doesn't it?

These attacks, and their consequences, will be demonstrated live, along with preventative measures that you can take to ensure they aren't happening on your cluster.

This talk explains CAP_NET_RAW and spoofing, but the audience is expected to be comfortable with Kubernetes concepts like pod specs and admission controllers.

The Windows Operating System does not support privileged operations from inside a container today. Daemon-sets on Windows nodes in Kubernetes clusters that need to perform configuration actions on the node are significantly impacted by the absence of privileged mode support on Windows. In this talk we:
1. Explore the pros and cons of the options the SIG Windows community brainstormed to provide containers running on Windows the ability to perform privileged operations while being managed by Kubernetes.
2. Delve into the specific characteristics of the privileged proxy approach we decided to adopt.
3. Demonstrate how the privileged proxy approach is used to support privileged operations that need to be executed by daemon-sets associated with CSI plugins running on Windows nodes in a Kubernetes cluster.

The explosion of Custom Resources in Kubernetes has lead to the development of new techniques to reference and reconcile objects in Kubernetes. Come learn how we are leveraging some simple patterns to produce complex systems within Kubernetes in the Knative project, and how you can adapt these methods to your applications.

Building a custom controller or operator to manage your Kubernetes applications is becoming easier, with the help of libraries and tools such as controller-runtime and Kubebuilder. Putting together an initial working prototype is fairly straightforward, but devil is in the details.

This talk focuses on lessons learned while writing Kubernetes controllers for stateful workloads with the help of controller-runtime. It covers some of the "hard parts".

The operator lives in the past: how to deal with resources cache inconsistencies? Why does idempotency matter? What can you do when StatefulSets are not good enough for the orchestration you need? How to empower advanced users but still provide good defaults? What namespace(s) should the operator have access to? How to test that monster you ended up building? These are questions engineers at Elastic had to answer.

Knative provides a way to extend Kubernetes to run serverless workloads. Although it runs as pods, given the nature of those workloads it requires an approach to security that is distinct from standard Kubernetes security practices. As 18th century explorers were wondering when they first encountered the platypus, is it a duck? an otter? or something else?
In this talk Ariel reviews the serverless threat landscape, which is quite differentiated from the container equivalent, using examples of how coding mistakes may expose applications despite the extremely ephemeral workloads.
This talk will show how combining preventative methods and more "offensive" methods such as tripwires can provide much better visibility and reduce the risk of Knative workloads being used as attack vehicles to reach other areas of the cluster or application.
Finally, the platypus question will be resolved.

Have you ever had to upgrade your Kubernetes clusters to update to a new release version, push new features or patch critical security vulnerabilities? Did it ever feel daunting to live update API masters or etcds? Can you automate such an operation?

We hope to share our musings at Lyft in solving the complexity of automating cluster upgrades and how that is incorporated into the design for - k8srotator - a Kubernetes custom controller.

As multiple components operating in cohesion make a cluster healthy, there are numerous points of failure that can occur during an upgrade cycle. Although there are varied ways of operating a Kubernetes cluster, the issues encountered during the process are common.

Attendees will walk away with knowledge about different cluster upgrade failures scenarios and ways to automate such operations without being in constant fear of losing the cluster state.

Uber relies on Big Data and ML to make business critical decisions such as pricing, trip ETA, etc. Today, those workloads such as Hive and Spark are running on YARN. To save millions of dollars by efficient use of cluster resources, Uber is planning to use Kubernetes to co-locate BigData/ML and micro-service workloads.

Kubernetes is the de-facto standard for running micro-services. However, in comparison to YARN, it still lacks many features like hierarchical resource pools, elastic resource sharing, gang scheduling etc. To bridge this gap, we have re-architected Peloton to be a set of Kubernetes scheduler and controller plugins so that we can provide feature parity with YARN.

This talk will cover:
- Learnings of running large-scale BigData/ML on Kubernetes with Peloton
- Colocation of mixed workloads
- Federation across zones
- Feature and API parity with YARN

A few months ago, I lost a whole day to debugging a problem with RBAC in Istio. I swore a lot, but I also learned a lot. I learned new tools, new interfaces, and the rabbit hole took me past most major parts of Istio.

Today I'll recreate that debugging session live, to show you the mental models and techniques I used to methodically follow this issue through a complex distributed system. We'll learn about systems debugging techniques in general, and operating Istio in particular.