Rotating secrets with serverless

Rotating secrets is a critical element to your security posture that, when done manually, is often overlooked due to it being a more and more tedious and complex process as the company and secrets grow. Security breaches due to secret access are difficult to detect as the hacker has gained access via legitimate means, and they can persist for months unless the secret is rotated. In order to do manual rotations developers have to keep track of when secrets need to be rotated, perform the process of rotating them, and update the application accordingly. Oftentimes teams have to plan rotation events every few months that involve going through a list of secrets and manually rotating each one. This is time consuming, tedious, and prone to human error, whether due to secrets being rotated incorrectly, or forgetting to add them to the list at all. In this article we are going to explore how we can use a serverless approach to automate the secret rotation process, avoiding having to ever endure one of these arduous events again!

For this article I will be using the example of rotating the keys for an AWS IAM service account, and updating them in a GitLab. Please understand this is very much a template. Individual parts can be swapped and replaced depending on your current stack and where your secrets are stored. In your situation you may be looking to rotate an API key, or a password, and update it in a database, or a VM. Whatever the situation, the parts may vary, but the concept from a serveless point of view are the same. The scope of this article is really just to give a broad overview of how the process works. For an in depth and technical explanation of the implementation, please take a look here. With that being said, let’s begin!

Overview

The fundamental concept of this process is based around 3 core parts:

1. A function to perform the rotation
2. A trigger as part of the service that needs the rotation done
3. Storage for the secret

In our example the function has an extra step that is required because of the way our service is structured, the trigger for each service passes the name of the IAM user whose keys are being rotated, and the secrets are stored in the GitLab project variables. The platform in this diagram merely represents whatever it is you are trying to access with the secret. In our case it would be AWS. I am using this as an example as it is one I have worked with first hand. There are definitely better ways to manage service account permissions and keys and inject them into your application, but this approach provides a simple and understandable method for what we are looking to achieve, as well as providing the opportunity to showcase how the core method can be modified (adding deployment), to provide a more holistic approach to secret rotation.

How to rotate your secret

The first thing we need to define is: How do I rotate my desired secret? In our example we are using IAM user security credentials. If we were doing this rotation manually we would find that user in the console, create new credentials for them, then use these new credentials to update the relevant variables in our GitLab project. Once updated we would then redeploy our service with the new values, and finally once deployed, we can remove the old key from the user so that it is no longer in circulation.

In order to translate this into our serverless function we will need to do this process via code. Once again, for technical details, please take a look here, however for the scope of this article, suffice to say the code was written in python and made use of boto3 for the AWS IAM elements, and python-gitlab for the GitLab elements. Please also be aware that your lambda function will need the correct permissions to make changes to IAM keys, as well as having a GitLab access token injected so that it has the ability to update variables there.

Triggering the function

When broken down into logical steps the function is straightforward to build and understand, however we need to be able to trigger it at the appropriate time for each service account that we have, so how do we achieve this?

So far we have created a stand alone “credential-rotator” function that performs a rotation, but on our desired service side we need to add functionality and infrastructure that will trigger the function with enough information to target the correct service account. To do this we simply create a Cloudwatch Event Rule. The rule runs on a schedule and triggers the lambda with an event that includes the name of the user being rotated and the GitLab project ID in which the variables need to be updated. Based on AWS best practices keys should be rotated every 90 days or less, so we can set this as the schedule. We can then repeat this process for each IAM user we wish to rotate, creating a system where when a key reaches its expiration it will automatically trigger the function to rotate it and deploy the service with the new keys.

This may seem daunting, but assuming you are using some form of IaC to deploy your code (which you should be!) you can create a “service account” module that packages this Cloudwatch Event Rule alongside the IAM user infrastructure so it can be easily included when creating new services, or updating those without the automatic rotation in place.

Summary

I have already mentioned it a couple of times, but it’s important so I’ll mention it again. The key here is not to get bogged down in the details of this specific example and instead take a broad view of how the process works so you can tailor it to your specific needs. Instead of GitLab and AWS IAM you can just as easily be updating API keys in a database to be used by containers in GKE, or updating terraform vault secrets to be used in future deployments. Whatever your setup, if you have secrets that are being rotated, you can do it automatically.