CI/CD pipelines are at the core of daily operations for many businesses today. These processes, when set up correctly, help to keep the delivery process consistent by automating many manual tasks and providing visibility into how the software is being worked on.
DevOps is also the place in your technology stack where your infrastructure has access to many different resources, from development and production environment to analytics keys and code signing credentials. The more resources your pipelines have access to (secure secrets, proprietary code, databases, etc.), the more important it is to keep your CI/CD system secure.
In this article, we talk about the different parts of DevOps security for CI/CD pipelines and highlight a number of ways to improve on these aspects. We usually divide security practices into three parts that you can address with different kinds of solutions:
- Secure pipeline configuration
- Code and Git history analysis
- Security policy enforcement
All three aspects are equally important. Let’s take a deeper look at these categories and ways to address them.
Secure Pipeline Configuration
It is possible to use your CI/CD pipeline configuration to make security issues less likely to happen.
First, safely store secrets that you use in your pipelines for connecting to databases and third-party services. Depending on the CI platform you use, you may have the option to use encrypted-at-rest environment variables, or to use the contexts feature. Contexts are used to provide access to environment variables across projects. Their use can also be restricted to specific security group members as defined by the organization’s administrator.
Another option is to use a third-party solution to dynamically fetch secrets from their secure storage for your jobs. To ensure that none of your credentials become exposed to those who shouldn’t see them, you should never check these into your repository in plain text, even if the repository is private.
Second, for sensitive files like code signing keys, I recommend adding an additional layer of isolation between your encrypted files and your repository for added security. Keep them encrypted in your repository and store the decryption key in the environment variables or contexts, only decrypting them inside your CI/CD jobs when they are needed. This makes it even more difficult to expose or leak these highly sensitive files.
Third, you never want to leave your CI/CD environment running without monitoring it. Therefore, make sure that the containers and VMs used for your pipelines are destroyed after jobs that require sensitive information finish running.
Fourth, pay extra attention to how your CI/CD system handles builds for pull requests that come from forks of your repository. If your build stages require secrets to work, forked pull requests might gain access to them through your pipelines.
By addressing these four aspects you can significantly reduce the risk of security incidents due to your CI/CD pipeline configuration. Next up, code and Git history analysis.
Code and Git History Analysis
We all love Git for its comprehensive history of changes, being able to look through the history of the project with a simple CLI command and see who has most recently changed a line of code, among other benefits. This also means that any sensitive information that makes it into the Git history can be accessible to attackers if a Git repository is exposed, even if the most recent state of the repository doesn’t contain the secrets anymore.
Trufflehog and GitLeaks are two tools that can help you identify secrets that have been committed to the codebase so that you can deactivate and replace them. These two tools will also scan your Git history for traces of secrets that your team might have added to the repository in the past.
Once your Git history is clear of secrets, you can proceed to the next level: making sure that your current revision doesn’t contain any vulnerable dependencies.
Static Application Security Testing (SAST) techniques can look through the application in your commit and analyze its dependencies. If any dependencies contain any issues or known security vulnerabilities, your commit will be marked as insecure and won’t be allowed to proceed to deployment.
Dynamic Application Security Testing (DAST) techniques go one step further and spin up a copy of your production environment inside your CI job in order to scan the resulting containers and executables. The dynamic aspect helps the system catch dependencies that are being loaded at launch time, for example, as those won’t be caught by SAST.
Security Policy Enforcement
Some security aspects can’t be statically checked based on known vulnerabilities, but rather are specific to your particular company, and therefore need to be codified as policies. These can take the form of automated or manual compliance checks.
Certain tasks such as reviewing the list of all accounts that have access to your repositories, or making sure your onboarding and offboarding processes are in sync, will be manual. We recommend that you set reminders to perform these regularly.
Some tasks actually can be automated quite easily. For example, third-party services can provide a convenient way to codify a set of rules that will be matched against in your CI pipeline, e.g., proving compliance with the regulations that govern your data. If there is a mismatch, the build will fail.
Some of the Important Parts Can’t Be Automated
Keeping your CI/CD pipelines secure is more important today than ever before as automating your CI/CD practices requires more access to development and production environments, analytics keys and code signing credentials than in the past. The hard thing about security is that if you only get one of the three categories right, you won’t be 33% secure. You will be 0% secure until you get all the bases covered.
With the approaches outlined above, you’ll be in a much more secure position than you would be without them. However, some of the aspects that matter the most for CI/CD security can’t be automated or detected with an automated policy. These aspects are shaped by the behavior and culture within your team.
A DevSecOps Mindset
Your team is your most valuable set of security researchers because they know your applications best and constantly work on them. Have you implemented a clear process for your teammates to report security issues if they find any, and do you dedicate time to fix the security flaws they find? Do you celebrate those who report issues openly? In many companies, the answer isn’t an immediate “yes” to these, and it should be.
How long does it take for someone on your team to report suspicious activity? Do they need to post in multiple Slack channels to get attention, or is there a streamlined process to get an issue like this looked into? How do you treat those who report false positives — do you blame them for wasting the team’s time, or do you hold a blameless post-mortem so that everyone on your team learns from the mistakes? All these questions can make or break your security-minded development culture.