KubeCon NA 2020 Key Takeaways: Platforms, Safety, and End Users
Highlights and interesting stories from the CNCF flagship event
The Ambassador Labs team and I have settled safely back into our daily routines after another excellent (virtual) KubeCon NA event last week. There were so many great sessions, and we all thoroughly enjoyed the hallway track and Slack channels discussions
Although no travel was involved this year there was still a lot of logistics to figure out, and so a massive tip of the hat to the entire CNCF KubeCon + CloudNativeCon team!
Here are our top five takeaways from the event:
- Platforms as product: Self-service is essential
- Safety (and speed): Move fast and don’t break things
- DevSecOps: Security is a day 0 concern
- Multi-cluster: The K8s future is connected
- End user focus: Learning and telling stories is important
Platforms as product: Self-service is essential
The concept of treating “platforms as product” was on display everywhere at KubeCon. Dave Sudia’s superb keynote summarised all of the key benefits and challenges of building an in-house platform. He argued that the CNCF landscape (and surrounding vendor ecosystem) is now mature enough to assemble an effective Kubernetes platform. However, if you are building rather than buying, this platform must be designed, managed, and operated just like any other product within an organization.
I’ve been lucky enough to have known Dave and the awesome GoSpotCheck team for a number of years, as they are users of Ambassador, and over this time I have learned a lot from him via his previous talks, podcasts, and general Slack banter.
Reinforcing Dave’s messages was Alena Prokharchyk in her excellent keynote, “The Cloud Native Journey @Apple”. Alena reiterated the benefits of taking the product approach to building a platform, and outlined a number of platform stakeholders from her experiences at Apple, from application developers, to SRE teams, to finance:
It was impressive to see the list of CNCF technology that Apple have embraced in their platform:
In addition to the focus on platform as product, many sessions and attendees emphasised the need for developer self-service for infrastructure. Dave Sudia also touched on this concept, reminding us that in the context of building platforms, application developers are the key customer and usability is a big part of the developer experience. Unless developers have the ability to speedily and safely configure how the platform runs their services — and also confiure how applications are released to end users — then the value of a cloud native platform will not be fully realised.
The always-informative Gene Kim ran an AMA at the Ambassador Labs booth (our “Ambassador Fest”), and he shared that one of the key metrics for any platform/ops team is how many “thanks you’s” they receive on a regular basis. Providing developers with self-service infrastructure is essential in order to deliver value effectively to them as users of the platform:
Safety (and speed): Move fast and don’t break things
There was a lot of interesting chatter around the benefits of using a GitOps practice to release applications with speed and safety in the event Slack channels.
Cornelia Davis’ talk was fantastic, “GitOps Is Likely More Than You Think It Is”, and a tip of the hat to Rich Burroughs for his usual excellent live tweeting skills:
Stephen Augustus’s keynote contained great storytelling about how the CNCF came to recognize the value of “releasing deliberately”, and correspondingly aimed to strike a balance between safety and speed with their projects. In 2020 the release cadence of the Kubernetes framework was reduced from four to three times a year:
Fault location in production was also a recurring theme, and implementing a solid (self-service) observability and alerting strategy was a presented as an essential practice. Make it easy for engineers to get access to metrics (ideally via a dashboard with good UX) and make it easy for them to ask ad hoc questions of the data (hat tip to Liz Fong-Jones and Charity Majors from Honeycomb.io for providing so much great advice in the Slack channels).
I also heard a lot about the benefits of using OpenTelemetry to consolidating the collection of metrics. The related keynote session by Constance Caramanolis was excellent. I captured the key takeaways from the talk here:
And my colleague, Alex Gervais, did a fantastic getting started Ambassador Fest breakout session on OpenTelemetry, using the K8s Initializer to bootstrap a playground to experiment with Lightstep and Jaeger:
I also wanted to give a big shout out to Ara Pulido from Datadog, who presented a great Ambassador Fest session on using techniques like distributed tracing and profiling to locate and triage production issues. Ara’s key advice was to bake observability into platforms and applications from day 0. Not only is this easier to implement and maintain with this approach, but it also saves time when the inevitable issues arise:
I was also impressed with the new Troubleshoot.sh project that the Replicated folks released at the event. This is a kubectl plugin that provides diagnostic tools for Kubernetes applications, which enables the codification of your support analysis. You can ship across your diagnostic and debugging steps to an end user, and without you having access to the cluster, you can still extract the info necessary for debugging. Dex Horthy dropped by our booth and presented a session on how to use the tool:
On a related topic, I chatted to a lot of folks about the CNCF Telepresence tool for local-to-remote cluster application debugging, and found myself shamelessly sharing the recording of my Telepresence 101 talk from KubeCon EU earlier in the year:
A new version of Telepresence is in the works, too, and we would love to hear your experience reports or your feedback on the tool:
Kenn Hussey, Ambassador Labs Director of Engineering, also presented a great booth session on the value of running game days (think fire drills for IT). This has really helped us focus on experimentation, safety, and learning from failure. Kenn shared some of the best practices the team has learned from running these over the years:
When you are working in the cloud, everything is running over the network, and so getting this configuration correct is essential for both speed (of release) and safety (for security and fault isolation). Ambassador Labs’ very own SVP of Engineering, Bjorn Freeman-Benson, was joined by Alyssa Wilk, Lin Sun, and Matt Klein to discuss “Tales from the Edge: Is the Edge More Important Than the Service Mesh”
Key takeaways from this panel were:
- Look for simpler architecture options [avoid complexity]
- Consider your requirements carefully [choose tech appropriately]
- Make friends early [in the OSS community]
I tried to live tweet as much of the wisdom being shared as possible, and the tweet below contains a long thread of goodies!:
A final shout out in this space goes to my colleague, Flynn, who is the lead engineer on the Ambassador API gateway. He did a great booth talk around scaling Ambassador (and Envoy) for safety, speed, and profit:
DevSecOps: Security is a day 0 concern
Over the past several KubeCons there has been an increasing focus on security. Ian Coldwater’s keynote in San Diego last year is still a standout moment in my KubeCon experiences. Ian was also present in this year’s keynote, and this time they brought along some friends for a super interesting SIG-Honk panel:
I also had several interesting chats about open policy agent (OPA), which I’ve been following for quite some time (after Gareth Rushgrove put me onto this).
The Styra team are doing great work in this space, as demonstrated in their talk, “Customizing OPA for a “Perfect Fit” Authorization Sidecar — Patrick East, Styra”. Their KubeCon session is well worth watching, and the good folks at The New Stack have also written a deeper dive in “Open Policy Agent for the Enterprise: Styra’s Declarative Authorization Service”.
In another security related topic, Peter ONeill presented a great Ambassador Fest session on a common challenge I bump into: understanding the integration of Amazon’s IAM for EKS and Kubernetes’ RBAC:
Multi-cluster: The K8s future is connected
Thomas Rampelberg and I presented a session “Multi-Cluster is Easier Than You Think with Linkerd and Ambassador”, which was a lot of fun. Many thanks to everyone who attended, and a big shout out to all the folks that submitted great questions. At the end of the talk and throughout the conference I had many interesting chats about multi-cluster use cases, experiences, and challenges.
Although we know of organizations that are operating multi-cluster service mesh configuration, most of the questions were coming from folks who were just starting dipping their toe in the water. Use cases included local to remote development (for fast feedback when working with dependencies — don’t forget Telepresence here, too), segmenting clusters/meshes for compliance (PCI-DSS etc), and limiting blast radius when upgrading K8s, the mesh, or other infra.
For the folks that asked about how to set up the K8s environments for the tutorial, I used Docker for Mac and Google Kubernetes Engine (GKE), which I bootstrapped with the K8s Initializer:
And the instructions for following along with the demo can be found here: https://linkerd.io/2/tasks/installing-multicluster/
If you have any further questions, then please send them across to Thomas or myself.
End user focus: Learning and telling stories is important
It was clear from the announcement of the event CFP right through to the end of the conference itself that the CNCF are focusing on two things: sharing end user stories and helping folks that are new to the cloud native landscape get started on their learning journey.
Diane Mueller’s great keynote set the scene for the benefits of end user participation with the CNCF and the wider cloud native sphere:
The GoSpotCheck and Apple keynotes were a great example of end user participation, as were the countless other sessions presented by organizations that have adopted Kubernetes. All of these speakers told fascinating stories of cloud adoption, and in my experience this is the key to getting others to follow in their footsteps.
My colleague, Kelsey Evans, talked about learning Kubernetes for non-engineers, and one of her key takeaways was not to be afraid to ask questions: the CNCF have worked really hard to foster a welcoming and inclusive community:
And on the keynote stage Cheryl Hung made a great case for the value of training and certification:
One company that stood out in regard to contributions to the end user ecosystem was Intuit — all of their sessions were excellent, both from a knowledge sharing and tool creation perspective — and they presented everything from continuous delivery and chaos with ArgoCD, a deep dive on Kustomize, and an experience report on machine learning with Kubeflow.
If you are an end user with a story to tell, I would thoroughly encourage you to submit to the virtual KubeCon EU 2021 (CFPs close on 13th December)
Until the next time we meet
It was great to virtually meet and chat with so many of you! Although I only spoke to a fraction of folks that I usually would in comparison to an in person event, the community spirit was still strong and I took away a lot of notes (a strong signal of a productive event for me!). Thanks to everyone involved: the organizers, speakers, attendees, and sponsors.
I’ll wrap up this post with a look to the cloud native future for 2021 and beyond, with a tip of the hat to Liz Rice for a great CNCF TOC keynote:
I also can’t write a KubeCon blog post without mentioning Dan Kohn’s legacy. The opening tributes to Dan from Priyanka Sharma and Cheryl Hung were truly heartfelt:
If you have any questions about anything I’ve written here, please feel free to reach out via Twitter @danielbryantuk or find me on the Ambassador Labs Slack.
