With the growing popularity of CI platforms to build software, bad actors are increasingly looking to exploit these environments to target organizations. In our post about the recent Codecov breach, we explored how an attacker was able to get access to credentials from within the CI/CD pipeline. To prevent this from happening, you need to shift left and secure your development lifecycle from the start. Using our open source tool Tracee can help you achieve this – let’s see how.
“Tracee-ing” threats as they happen
Tracee is Aqua’s open source runtime security project designed to trace your system and applications at runtime, analyze the collected events, and detect suspicious behavioral patterns. Apart from protecting production environments, Tracee can also be used for CI/CD pipeline security, leveraging the same technology in a different setting. This is mostly possible due to GitHub Actions’ ability to run privileged eBPF programs in the pipeline and its flexibility for writing workflows like ordinary Linux scripts.
1 – Creating a demo app
In this post, I will explain how you can start protecting your CI/CD pipelines against supply chain attacks. Let’s start by coding a simple app that does nothing but prints to standard output.
2 – Deploying on GitHub Actions
Next, we’ll create a basic GitHub Actions workflow. GitHub Actions is freely available for public projects.
This simple YAML will checkout, build and run our simple app inside of an Ubuntu environment.
3 – Adding Tracee
Next, we’ll add Tracee to this CI/CD pipeline with one easy step:
In this example, we create a simple pipeline that runs Tracee as a service container in the background. GitHub Actions ensures that service containers are created and available before the CI/CD pipeline gets executed so we can be confident that Tracee will capture all events in our pipeline.
In order to see Tracee’s logs, we’ll add a small stanza to our pipeline as shown below:
4 – Running the CI pipeline
Now let’s see what happens when we trigger the CI/CD pipeline:
As expected, our code runs and prints to standard out. We can also see the output of Tracee logs, which means the tool is ready to detect any malicious activity. By default, Tracee comes out of the box loaded with a handful of common signatures which help detect malicious behavior. You can find the source code for all these signatures in our GitHub repo.
5 – Making our app delimalicious-ish
Now to the fun part. Let’s add some malicious behavior to our demo app:
Here, we’re calling the new function but inside that function, we shell out another Python script, which looks like this:
The Python script creates a socket and redirects to stdin. This potentially can be exploited by a remote adversary to communicate with the running pipeline.
6 – Observing the logs
We check this code in and run our CI/CD pipeline once again. This time the output looks a little different.
As you can see, Tracee was able to pick up this unwarranted invocation of poc.py and flag it as a suspicious event. Specifically, it was detected by TRC-1, the Standard Input/Output Over Socket signature of Tracee.
7 – Automating actions on events
With the event detection in place, it’s easy to set up a failing pipeline step in case something suspicious is found in your CI:
This will return a non-zero exit code if Tracee detects any malicious behavior, and the build pipeline will fail.
What’s next?
As CI workloads increase, it is crucial to embed security into your development lifecycle right from the start. Tracee can help you “shift left” and secure your applications at build by detecting malicious activity in the CI pipeline in real time. Soon, we’ll be releasing the official Tracee version for GitHub Actions – stay tuned!
You can find Tracee in our GitHub repository.
Learn how Aqua DTA can help you protect against supply chain attacks.