The Kubernetes project has disclosed a super-critical security bug that’s in every supported version (and probably loads of unsupported ones, too).
Patches are available, unless you’re on an unsupported version (entirely possible—the oldest supported upstream version came out about nine months ago).
Time to panic? In this week’s Security Blogwatch, we pitch a patch.
Your humble blogwatcher curated these bloggy bits for your entertainment. Not to mention: Little green men …
K8S h8s API vuln
What’s the craic? Steven J. Vaughan-Nichols—Kubernetes' first major security hole discovered:
[It’s] the most popular cloud container orchestration system by far, so it was only a matter of time. … CVE-2018-1002105, aka the Kubernetes privilege escalation flaw, is a doozy.
It's a CVSS 9.8 critical security hole. … Any user can establish a connection through the Kubernetes … API server to a backend [and] send arbitrary requests … authenticated with the Kubernetes API server's … TLS credentials.
…
Yes, anyone who knows about this hole can take command of your Kubernetes cluster. … Fortunately, there is a fix, but some of you aren't going to like it.
…
If you're still using Kubernetes v1.0.x-1.9.x, stop. … The only real fix is to upgrade Kubernetes.
…
As far as anyone knows, no one has used the security hole to attack anyone yet. … But abusing the vulnerability would have left no obvious traces in the logs.
Can it be true? Thomas Claburn calls it a cluster-fact:
Kubernetes version v1.10.11, v1.11.5, and v1.12.3 have been made available to fix … a privilege escalation vulnerability. … There are two primary attack vectors.
Using the first, an individual [with] privileges granted to a normal user by default can become a cluster-admin. … The second method lets an unauthenticated user access the API to create unapproved services.
…
The API server is the main management entity in Kubernetes. It talks to the distributed storage controller [and] the agents overseeing each node in a cluster of software containers.
Swimming upstream? Jordan Liggitt announces an Announcement:
Kubernetes v1.10.11, v1.11.5, and v1.12.3 have been released to address … a critical security issue. … We recommend all clusters running [any] previous versions update to one of these releases immediately.
…
Thanks to Darren Shepherd for reporting this problem.
How the heck did he find that? Darren Shepherds the story: [You’re fired—Ed.]
I discovered this issue in early November, but the bug actually goes back a couple years and is a pretty interesting story. [But] if you are running a hard multi-tenant cluster (untrusted users) you should be concerned.
…
This CVE shows how strong the community is and how well run it is. … Only while fixing this bug did we discover it had security implications. I filed the issue to the Kubernetes community through their established security disclosure process and very quickly the issue was properly patched upstream and backported to 1.10, 1.11, 1.12, and 1.13.
…
Take comfort knowing that users of Kubernetes are in good hands. Don’t panic, just upgrade your clusters and carry on.
Wait. Pause. What about older versions? So asks coleca:
Shame they aren't updating anything older than 1.10. 1.09 was released just a year ago.
The commercial K8S vendors seem to be doing the patch all the way back. Smart move by them to signal to the enterprise the value of using a commercially supported K8S distribution over something like kops or kubeadm.
And falcolas agrees:
Move Fast and Break Things has broken our industry in many fundamental ways. Just moving from 1.10 to 1.13 will result in broken integrations due to changed/deprecated APIs, no matter that it's a minor version change.
But Andrew Stuart disagrees:
If you're using something like kops or kubeadm, chances are you're on the latest anyway and updating will take you all of an hour and zero downtime.
The Kubernetes project is pretty explicit about only supporting 3 minor versions back, which gives you a full 3 quarters to figure out if the changes break anything you have deployed and fix those cases. If that's too fast, you're probably an Enterprise anyway and uncomfortable with anything but vendor support.
How did we get here? James Orme explains:
Yes, you heard it right: … Any sensitive data can be stolen, malicious code injected, heck, if a hacker fancied it, they could just terminate an application altogether – all from within the … firewall. Yikes.
…
Find any firm at the forefront of digital transformation and there’s one thing you can bet on: … It’s using Kubernetes to orchestrate containerized applications – enabling incredibly complex composite services comprising simpler microservices.
…
A Kubernetes craze has taken hold of the cloud community, and its rapid adoption has left … Docker, eating dust. … Kubernetes won the container war sometime in 2017.
…
[But this] “privilege escalation flaw” is a nasty piece of work.
Wake up and smell the coffee. Kris Nova—@krisnova—is fresh as an Indonesian spiritual dagger:
Couldn't sleep so started reading the Kubernetes CVE. … This was a good one :D
…
You have to access to the api server and we’ve been moving away from the aggregated api model for a while now. But yeah using the api tls is like basically having root on a kubelet.
…
I honestly wish we could submit a talk on this at KubeCon. … Almost as much as I wish we would document the kubelet api server and its relationship with the api server.
…
All I’m saying is this is bad, but it’s easy to fix. Also we probably ought to review how the api server auths w/ the kubelet in general, and shedding light on these arch decisions starts with docs. Most people don’t even know the kubelet has an api server let alone what it does.
…
this is a great exploit and there’s a lot of good teaching points along the way.
And Andrew van der Stock agrees, via Tara Seals:
APIs can be difficult to test by traditional security testing tools and approaches, and to a certain extent, the security industry has not kept up, primarily because most are not developers themselves. … As a whole, the security industry needs to shift left, adopt the same tooling as developers, and write unit and integration tests that fully exercise APIs, particularly those that have the potential to alter the state of an application or extract bulk personal information.
…
APIs make the friction of doing business much less. … However, whilst there are new risks to APIs not covered by previous applications, application security is near universal and still is incredibly relevant. … Securing APIs should be the focus of every organization that uses them.
Meanwhile, here come the predictable Luddites, such as the irrepressible Mike P UK:
Cloud computing and storage has always been insecure and this proves it. I will NEVER use the cloud for anything. If a software application requires its use then I will not use that software, simple.
The moral of the story?
Always be ready to patch—at a moment’s notice. And ensure your infrastructure is still using supported versions.
And finally …
You have been reading Security Blogwatch by Richi Jennings. Richi curates the best bloggy bits, finest forums, and weirdest websites … so you don’t have to. Hate mail may be directed to @RiCHi or sbw@richi.uk. Ask your doctor before reading. Your mileage may vary. E&OE.
Image source: Carla Wosniak (cc:by)
Keep learning
Learn from your SecOps peers with TechBeacon's State of SecOps 2021 Guide. Plus: Download the CyberRes 2021 State of Security Operations.
Get a handle on SecOps tooling with TechBeacon's Guide, which includes the GigaOm Radar for SIEM.
The future is security as code. Find out how DevSecOps gets you there with TechBeacon's Guide. Plus: See the SANS DevSecOps survey report for key insights for practitioners.
Get up to speed on cyber resilience with TechBeacon's Guide. Plus: Take the Cyber Resilience Assessment.
Put it all into action with TechBeacon's Guide to a Modern Security Operations Center.
