Setting Up Cross Realm Trust with Kerberos KDC

In this blog we will show you how our Gitlab AWS credential helper will make it very easy to connect to an AWS account using the gitlab pipeline id token as your identity.

Introduction

Since June 2022, Gitlab provides JWT identity tokens for each project pipeline, which can be exchanged for AWS access credentials using the assume-role-with-web-identity API call. This is great, because it allows you to configure the AWS permissions for each individual pipeline. However the problem is that exchanging the token for credentials requires quite a bit of shell programming. Something along the following lines:

ROLE_ARN=$1
WEB_IDENTITY_TOKEN=$2
ROLE_SESSION_NAME=gitlab-${CI_PROJECT_PATH_SLUG}-${CI_PIPELINE_ID}

CREDENTIALS=($(aws sts assume-role-with-web-identity \
        --role-arn ${ROLE_ARN} \
        --role-session-name ${ROLE_SESSION_NAME} \
        --web-identity-token "${WEB_IDENTITY_TOKEN}" \
        --duration-seconds 3600 \
        --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' \
        --output text))

aws configure set aws_access_key_id "${CREDENTIALS[0]}"
aws configure set aws_secret_access_key "${CREDENTIALS[1]}"
aws configure set aws_session_token "${CREDENTIALS[2]}"

As you can see that is quite a complex shell script. This will not look pretty in your .gitlab-ci.yml, and you will would probably add this as a separate script to your project. For a single project this is ok-ish, but to apply that to all your projects, is not going to scale well: As this script is not suitable for all situations, you probably end up with some variants as well.

Gitlab AWS credential helper to the rescue

This is where the gitlab aws credential helper comes in! The utility is flexible and very easy to use in all sorts of situations. It is simple to configure, as it only needs the AWS account number and Gitlab id token. The Gitlab project path slug and pipeline id will be used to determine the IAM role- and session name. The general configuration of the CI/CD pipeline consists of the following five steps:

Add the Gitlab ID token
Set the AWS variables
Configure the credential helper
Create the IAM role for the pipeline
Install the credential helper

Let’s us take you through these five steps in the following paragraphs.

Add Gitlab ID token

To obtain an Gitlab ID token in your pipeline jobs, add the following definition to your pipeline definition:

default:
  id_tokens:
    GITLAB_AWS_IDENTITY_TOKEN:
      aud: https://gitlab.com

This will cause each job to have an environment variable name GITLAB_AWS_IDENTITY_TOKEN with the JWT identity token signed by Gitlab as a value. The credential helper expects the token in this variable.

Set AWS variables

To know which AWS account to connect to, add the environment variable GITLAB_AWS_ACCOUNT_ID with your AWS account id:

variables:
  GITLAB_AWS_ACCOUNT_ID: '1234567898012'

If you have many gitlab projects, you can also set this variable as a CI/CD variable on the group level. We recommend to add the following variables as well:

variables:
  - AWS_CONFIG_FILE: ${CI_PROJECT_DIR}/.aws/config
  - AWS_SHARED_CREDENTIALS_FILE: ${CI_PROJECT_DIR}/.aws/credentials
  - AWS_PROFILE: default

These will make sure the credential configurations are kept in scope of the pipeline build and that the obtained credentials are actually used.

Using the credential helper

Now you are ready to use the credential helper, in one of three different ways:

as credential process
stored as shared credentials
as environment variables

Let’s look at each of these options.

As credential process

The most elegant use, is to configure the credential helper as AWS credential process, like shown below:

before_script:
  - >
    aws config --profile default 
    set credential_process 
    "gitlab-aws-credential-helper process"

script:
  - aws --profile default sts get-caller-identity

The AWS SDK will call the credential helper whenever it requires to access to AWS!

Stored as shared credentials

Alternatively, the helper can store the AWS credentials directly in the AWS shared credentials file. You can do this as follows:

before_script:
  - >
    gitlab-aws-credential-helper aws-profile 

script:
  - aws --profile default sts get-caller-identity

This will store the credentials in shared credentials file, normally ~/.aws/credentials.

Get as environment variables

Finally, you can use the helper to export the environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
and AWS_SESSION_TOKEN, as shown below:

script:
  - eval $(gitlab-aws-credential-helper env --export)
  - aws sts get-caller-identity

Alternatively, you can directly execute the command:

script:
  - gitlab-aws-credential-helper env -- aws sts get-caller-identity

Create the pipeline IAM role

Once the pipeline is prepared, you need to add the IAM role for the CI/CD pipeline. The name of the role should be equal to the name of the gitlab project path slug as defined by the variable CI_PROJECT_PATH_SLUG, prefixed with gitlab-. The following CloudFormation template shows an example IAM role definition:

AWSTemplateFormatVersion: '2010-09-09'
Description: Role for gitlab pipeline with limited AWS access
Parameters:
  RepositoryPath:
    Description: the gitlab repository path
    Type: String
  RepositoryPathSlug:
    Description: the gitlab repository path slug
    Type: String
Resources:
  GitlabPipeline:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub 'gitlab-${RepositoryPathSlug}'
      MaxSessionDuration: 3600
      AssumeRolePolicyDocument:
        Statement:
          - Action: sts:AssumeRoleWithWebIdentity
            Principal:
              Federated: !Sub 'arn:aws:iam::${AWS::AccountId}:oidc-provider/gitlab.com'
            Effect: Allow
            Condition:
              ForAnyValue:StringLike:
                "gitlab.com:sub":
                  - !Sub "project_path:${RepositoryPath}:ref_type:branch:ref:*"
      Policies:
        - PolicyName: MetaInformationAccess
          PolicyDocument:
            Statement:
              - Effect: Allow
                Action:
                  - sts:GetCallerIdentity
                Resource:
                  - '*'

The template assumes you have already created the OIDC provider for gitlab.com in your account. If not, checkout this blog.

To create the role you need, type:

CI_PROJECT_PATH=mvanholsteijn/aws-credential-helper-demo
CI_PROJECT_PATH_SLUG=mvanholsteijn-aws-credential-helper
aws cloudformation deploy \
  --stack-name gitlab-$CI_PROJECT_PATH_SLUG \
  --template-file gitlab-pipeline-limited-access.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
   --parameter-overrides \
     "RepositoryPath=$CI_PROJECT_PATH" \
     "RepositoryPathSlug=$CI_PROJECT_PATH_SLUG"

This all you need for one pipeline. The following two paragraphs will explain how to install the credential helper and show you how to override defaults.

Install the credential helper

Of course, your job will need to have the gitlab-aws-credential-helper installed. We recommend to add it to the image that runs your job using a multi-stage docker build file, as show below:

FROM ghcr.io/binxio/gitlab-aws-credential-helper:0.3.2 as gitlab-aws-credential-helper

FROM <yourimage>

COPY --from=gitlab-aws-credential-helper /usr/local/bin/gitlab-aws-credential-helper /usr/local/bin

If you do not manage your own runner image, you can add the following command to the before_script section.

NAME=gitlab-aws-credential-helper
RELEASE=0.3.2
URL=https://github.com/binxio/$NAME/releases/download/${RELEASE}/${NAME}_${RELEASE}_linux_amd64.tar.gz
CHECKSUM=b8c84f1ca5622f4b0f529eca74e7689ea20418eab31a96666366ae1e1531b41f
ARCHIVE="$(basename $URL)"

cd /tmp && \
    curl -L -O -sS $URL && \
    (echo "$CHECKSUM  $ARCHIVE" | sha256sum -c -) && \
    tar -C /usr/local/bin/ -xzf $ARCHIVE && \
    rm -f $ARCHIVE

Overriding defaults

The credential helper uses sensible defaults to make it as easy as possible. Of course, you can override all options via environment variables or via the command line. The following example shows how to create different profiles for a named role in different accounts:

before_script:
  ->
    gitlab-aws-credential-helper
    aws-profile --name development
    --aws-account 111111111111 
    --role Deployer
  ->
    gitlab-aws-credential-helper
    aws-profile --name test
    --aws-account 222222222222 
    --role Deployer
  ->
    gitlab-aws-credential-helper
    aws-profile --name production
    --aws-account 333333333333 
    --role Deployer

Use the --help flag to get an overview of all possible options for each of the command usages.

Conclusion

With the Gitlab AWS credential helper, it becomes really easy to provide each Gitlab CI/CD pipeline with the least amount of privileges required to perform their task on AWS. Just configure the AWS account number and add the credential helper to the runner and create an IAM role for each gitlab project.

Seamlessly connect Gitlab CI/CD with AWS: Gitlab AWS credential helper