Kubernetes’ role-based access control (RBAC) system is a cornerstone of cluster security. Most clusters use RBAC to determine which users have access to specific operations, and its core elements are well covered in the Kubernetes documentation. However, there are some less well-known features that could be relevant when creating or using tools designed to ensure cluster security. A notable example is virtual verbs.
In general, because Kubernetes RBAC is REST-based, it maps HTTP verbs to permissions. For example, POST is mapped to the create right. However, there are some cases where RBAC knows about verbs that don’t map in this way. To understand that, we need to talk about “virtual verbs” and how the Kubernetes authorization model evaluates requests.
Although most operations do follow the REST conventions, Kubernetes’ authorization system is much more flexible than that. In essence, it looks at the strings present in all the RBAC roles and cluster roles in the cluster and does an equality check against the verb, object, and namespace requested. If there’s a match, the authorization check passes, and if not, it fails.
This is a bit abstract, so let’s create an example based on Jordan Liggitt’s Effective RBAC talk from KubeCon North America 2017. During this talk, Jordan demonstrated how RBAC works by using the example of a pirate who wanted to be allowed to educate dolphins in addition to his other duties.
So let’s use the example of a Kubernetes cluster that has been extended with dolphin education functionality by installing some new software. This software wants to leverage Kubernetes’ existing authorization framework to make sure that only authorized dolphin trainers can carry out this duty. To do this, RBAC will need to understand dolphin objects and the educate verb.
Teaching Kubernetes to educate dolphins
Kubernetes provides a handy mechanism to ask questions of the authorization system, with kubectl auth can-i statements. Doing this with a standard kubeadm cluster, we can ask if I’m allowed to educate dolphins. Running it as a cluster-admin user looks like this:
kubectl auth can-i educate dolphins
Warning: the server doesn't have a resource type 'dolphins'
Warning: verb 'educate' is not a known verb
yes
There are a couple of interesting things in this output. First, we get an expected error, which is that the server doesn’t have a resource type dolphins. It’s not one of the base Kubernetes objects. Next, we get an error that educate is not a known verb — however, then it ends with “yes”.
The reason for this is that cluster-admin rights are “*” to all resources for all verbs. Because * is a wildcard, it matches everything. So if you ask, “Does the string educate match *,” the answer will always be yes.
To show the negative case, we’ll ask if the default service account in the default namespace can educate dolphins:
kubectl auth can-i --as=system:serviceaccount:default:default educate dolphins
Warning: the server doesn't have a resource type 'dolphins'
Warning: verb 'educate' is not a known verb
no
The output is mostly the same, but at the end the answer is no, because this service account doesn’t have any wildcard rights and has no explicit rights for this operation.
Adding dolphins to Kubernetes
Our first step is to add a dolphin object to Kubernetes. We can do this easily, since Kubernetes is a flexible system designed for extensibility, by using custom resource definition (CRD).
This CRD will add a new object type dolphin.ship.io with a short name of dolphin:
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: dolphins.ship.io
spec:
group: ship.io
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
name:
type: string
scope: Namespaced
names:
plural: dolphins
singular: dolphin
kind: Dolphin
shortNames:
- dolphin
categories:
- all
After applying it to our cluster, we can re-run our test:
kubectl auth can-i educate dolphins
Warning: verb 'educate' is not a known verb
yes
We’ve now gotten rid of the error about there being no dolphins! This works fine for our cluster-admin user with their wildcard rights, but to add this specific right to the default service account, we’ll need a role to add this permission:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: dolphin-educator-role
namespace: default
rules:
- apiGroups:
- ship.io
resources:
- dolphin
verbs:
- educate
Applying this role to our cluster, we can see that it’s accepted even though Kubernetes doesn’t know anything about educate as a verb.
This is an important point about Kubernetes’ authorization design. Because it doesn’t have a fixed list of permitted verbs, if you make a typo in a verb or resource statement, it can be hard to debug because the server will just accept it.
After applying this role and a rolebinding to connect it to the default service account, we can list the permissions of our default service account and see that we now have rights to educate dolphins!
kubectl auth can-iauth can-i --as=system:serviceaccount:default:default --list
Resources Verbs
selfsubjectaccessreviews.authorization.k8s.io [create]
selfsubjectrulesreviews.authorization.k8s.io [create]
dolphin.ship.io [educate]
To actually have this verb do something would require more work. Specifically, you’d need a custom controller that used RBAC permissions and understood how to educate dolphins.
Virtual verbs in core Kubernetes
Assuming your cluster isn’t nautically inclined, where might you actually encounter these virtual verbs? In base Kubernetes, there are a couple of places where they occur, and these can be important from a security standpoint.
- Impersonate: This verb can be used in RBAC rules to refer to users and groups and allows sudo-like functionality
- bind and escalate: These verbs are applied to roles and clusterroles in Kubernetes and allow for privilege escalation by users who can use them.
Conclusion
Now that we know about virtual verbs in Kubernetes RBAC, what does it mean for our clusters? There are several takeaways to keep in mind:
- Tooling that evaluates RBAC rules may need to account for there being dynamically created resources in clusters.
- It’s important to include the known virtual verbs in tooling as well, or you could miss security critical escalation points.
- Be careful of typos when constructing RBAC rules. Kubernetes can’t warn you if you get something wrong, because it doesn’t have a list of what “right” looks like.