Part 2: API Access Control and Authentication with Kubernetes, Ambassador and ORY Oathkeeper: Q&A and Tutorial

Published in

Ambassador Labs

12 min readSep 5, 2018

The Ambassador Labs and ORY teams have recently been discussing the challenges of API access control in a cloud native environment, the highlights of which I captured below in a Q&A. There are many possible solutions (with associated benefits and tradeoffs), and our friends at ORY have put together a tutorial on how to use their Oathkeeper with Ambassador running on Kubernetes, which I have also included below.

API Access Control and Authentication with Kubernetes — Introduction

The web application and web service landscape is changing radically as large software companies are making their internal infrastructure and software development and operation practices open to the public. Initiatives such as the Cloud Native Computing Foundation, open source standards, and software like Istio and Kubernetes, make a big impact on how software is developed and operated. Go, the programming language written and maintained by Google shines with its toolchain. However, some of the tools behave differently than expected, and it may cost you several hours of debugging and experimenting to find the arguments and execution orders.

This also affects access control — which many developers have a love-hate relationship with. In the past, we have relied on language-level APIs provided by libraries such as OmniAuth, Spring Security, and PassportJS. These libraries will always have their place in the developer’s toolbox. However, as applications grow and companies move away from monoliths to the Service Mesh, using these libraries isn’t quite so easy anymore.

As you move to a distributed service architecture, you move away from integrating with local libraries and SDKs and calling services that operate on the network. This happens naturally as you adopt more languages (e.g., using the best language for each use case) and start more services. This, combined with differing zones of trust across your network, obviously impacts how you perform access control.

Q&A: Aeneas Rekkas, CEO of ORY, on the Challenges of “Zero Trust” Access Control

To set the scene, I recently sat down with Aeneas Rekkas, founder and CEO of ORY, and explored the concepts mentioned in more detail:

Ambassador Labs: Can you explain what you mean by “Zero Trust API Access Control”?

Aeneas Rekkas: Companies usually differentiate between an internal network (intranet) and an external network (internet). The intranet is typically within the company’s physical premises (e.g., office), and most, if not all, traffic from within that network is trusted without question. This is, for apparent reasons, a bad security practice. Advanced persistent threats (APTs) reek havoc in such environments as, once they are in the network, they comprimes everything else. Another issue is the rise of remote work and using your own device. Both do not play well with giving internal and external traffic different privileges.

Zero Trust API Access Control defines that each API is protected, regardless of where the traffic comes from. Instead of having a file server API that anyone within the intranet can access, the file server should instead require valid credentials. These credentials should typically come from one source (Authorization Server) and could be API Keys, Bearer Tokens, TLS Client Certificates, or HTTP Basic Authorization. Because this system will be deployed with every service, it is good to use an open source service specifically designed for this use case, such as ORY Oathkeeper.

Ambassador: What are the advantages of using ORY over other solutions?

AR: We started ORY because developer tools are clunky, and so is application security. ORY’s vision is to improve the way developers approach application security with an open source ecosystem. All services are written cloud-first and adhere to principles like 12-factor design. Besides extremely low resource consumption, every service deploys in seconds due to a ~12MB docker image size. The ORY ecosystem is being used in production by SME and F200 alike and has been acclaimed for its straight-forward design.

Ambassador: Why did you choose to integrate with the Ambassador API Gateway? What has your experience working with Ambassador been like?

AR: Ambassador is the perfect vehicle to implement Zero Trust API Access Control on top of. With its Kubernetes-first approach, it perfectly fits in the Istio Service Mesh. Because we don’t want to reinvent the wheel and write another API Reverse Proxy, integrating with Ambassador was a no-brainer. The community and maintainers from DataWire are exceptionally responsive and helpful. You should give it a try!

Ambassador: What are the future plans for ORY?

AR: We’re just getting started. Besides working on a new open source Identity Management product called ORY Hive, we are planning a cloud environment service for running the ORY Ecosystem with zero friction and managed security. Our vision is to make cutting-edge application security available as AWS EC2 or GCP Compute Engine!

Tutorial: API Access Control with Ambassador and Oathkeeper

Let’s try and make some of the concepts discussed a little more concrete by setting up an example with Ambassador and ORY Oathkeeper on Kubernetes. Before you go ahead, you’ll need to:

Make sure you have access to Kubernetes — either via minikube, Docker Desktop, managed Kubernetes, or any other type of Kubernetes deployment.
Make sure kubectl is configured and pointed to your Kubernetes deployment.
Download ORY Oathkeeper CLI and put it in your PATH.
On Mac or Linux, you will need to make the binary executable (and you may also want to rename it to something more convenient): $ mv oathkeeper-darwin-amd64 oathkeeper && chmod u+x oathkeeper

Ambassador Edge Stack API Gateway

Ambassador is a Kubernetes-native API Gateway built on the Envoy Proxy. Ambassador supports various features needed in an edge proxy, e.g., rate limiting, distributed tracing, dynamic routing, metrics, and more. Ambassador also includes an authentication API where you can plug in an external authentication service. This is the API that we will be using in this post.

Deploying & Configuring Ambassador Edge Stack

The first step is confirming that kubectl is set up properly:

$ kubectl get service kubernetes
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   21m

I highly recommend completing the Ambassador Getting Started tutorial.

To deploy Ambassador in your default namespace, first you need to check if Kubernetes has RBAC enabled:

$ kubectl cluster-info dump --namespace kube-system | grep authorization-mode

If you see something like --authorization-mode=Node,RBAC in the output, then RBAC is enabled.

Note: If you’re using Google Kubernetes Engine with RBAC (which is the default for all new clusters), you will need to grant permissions to the account that will be setting up Ambassador. To do this, get your official GKE username, and then grant cluster-admin role privileges to that username:

$ kubectl create clusterrolebinding my-cluster-admin-binding --clusterrole=cluster-admin --user=$(gcloud info --format="value(config.account)")

If RBAC is enabled:

$ kubectl apply -f https://getambassador.io/yaml/ambassador/ambassador-rbac.yaml

Without RBAC, you can use:

$ kubectl apply -f https://getambassador.io/yaml/ambassador/ambassador-no-rbac.yaml

Defining the Ambassador Service

Ambassador is deployed as a Kubernetes service. Create the following YAML and put it in a file called ambassador-service.yaml.

---
apiVersion: v1
kind: Service
metadata:
  name: ambassador
spec:
  ports:
    -
      port: 80
  selector:
    service: ambassador
  type: LoadBalancer

Deploy this service with kubectl:

$ kubectl apply -f ambassador-service.yaml

The YAML above creates a Kubernetes service for Ambassador of type LoadBalancer. All HTTP traffic will be evaluated against the routing rules you create. Note that if you're not deploying in an environment where LoadBalancer is a supported type, you'll need to change this to a different type of service, e.g., NodePort.

Creating your first route

Create the following YAML and put it in a file called httpbin.yaml:

---
apiVersion: v1
kind: Service
metadata:
  annotations:
    getambassador.io/config: |
        ---
        apiVersion: ambassador/v0
        kind:  Mapping
        name:  httpbin_mapping
        prefix: /httpbin/
        service: httpbin.org:80
        host_rewrite: httpbin.org
  name: httpbin
spec:
  ports:
    -
      name: httpbin
      port: 80

Then, apply it to the Kubernetes with kubectl:

$ kubectl apply -f httpbin.yaml

When the service is deployed, Ambassador will notice the getambassador.io/configannotation on the service and use the Mapping contained in it to configure the route. (There's no restriction on what kinds of Ambassador configuration can go into the annotation, but it's important to note that Ambassador only looks at annotations on Kubernetes services.)

In this case, the mapping creates a route that will route traffic from the /httpbin/endpoint to the public httpbin.org service. Note that you are using the host_rewrite attribute for the httpbin_mapping — this forces the HTTP Host header, and is often a good idea when mapping to external services.

Testing the Mapping

To test things out, you'll need the external IP for Ambassador (it might take some time for this to be available):

$ kubectl get svc -o wide ambassador

Eventually, this should give you something like:

NAME         CLUSTER-IP      EXTERNAL-IP     PORT(S)        AGE
ambassador   10.11.12.13     35.36.37.38     80:31656/TCP   1m

You should now be able to use curl to httpbin:

$ curl 35.36.37.38/httpbin/ip
{
  "origin": "< your IP address >"
}

or on minikube:

$ minikube service list
|-------------|----------------------|------------------------------|
|  NAMESPACE  |         NAME         |             URL              |
|-------------|----------------------|------------------------------|
| default     | ambassador           | http://192.168.178.108:32548 |
| default     | ambassador-admin     | http://192.168.178.108:30428 |
|-------------|----------------------|------------------------------|

$ curl http://192.168.178.108:32548/httpbin/ip
{
  "origin": "< your IP address >"
}

When you have found your Ambassador IP, I would recommend placing this into an appropriate variable e.g.

$ export AMBASSADOR_IP=192.168.178.108:30428

ORY Oathkeeper

ORY Oathkeeper is a cloud native Identity & Access Service. It evaluates incoming HTTP requests based on a set of rules, decides whether the request should be allowed or not, and converts the session data to a consumable format. Decisions are made by consulting two deciders: Authenticators and Authorizers.

Authenticators look for access credentials in the HTTP header, such as the bearer token, and implement business logic that validates those credentials. ORY Oathkeeper currently ships with different authenticators:

The JWT authenticator looks for the bearer token in the HTTP header and treats the value as a JSON Web Token. You can define which signature verification algorithm (HS256, RS256, …) should be used and provide the required key(s).
The OAuth 2.0 Token Introspection authenticator extracts the bearer token from the HTTP header and performs the OAuth 2.0 Token Introspection flow. This authenticator works great with ORY Hydra!

For a complete list of implemented authenticators, head over to the ORY Oathkeeper developer guide.

Authorizers use the session state returned by the authenticator to authorize the request. This could be by consulting an Access Control List (ACL), Role-Based Access Control (RBAC), or more advanced Access Control Policy Definitions like the one provided by ORY Keto.

Credential Issuers convert the session state returned by authenticators to an easily consumable format. The session state can be converted to a JSON Web Token signed with a private/public keypair, HTTP Headers, and HTTP Cookies.

ORY Oathkeeper has two operational modes. One is a reverse proxy that can be deployed as a sidecar or in close proximity to the API Gateway. The second is as anAPI which is connected to the API Gateway of your choice. For this tutorial, you will exclusively look at the API operation mode.

Deploying and Configuring ORY Oathkeeper

First you need to create a secret which will be used to sign the ID Token. The secret must be 32 characters long:

$ kubectl create secret generic ory-oathkeeper --from-literal=CREDENTIALS_ISSUER_ID_TOKEN_HS256_SECRET=<your-secret>
# For example:
# $ kubectl create secret generic ory-oathkeeper --from-literal=CREDENTIALS_ISSUER_ID_TOKEN_HS256_SECRET=dYmTueb6zg8TphfZbOUpOewd0gt7u0SH

Next, deploy the ORY Oathkeeper Service and Deployment in “API mode”.

$ kubectl apply -f https://raw.githubusercontent.com/ory/k8s/master/yaml/oathkeeper/simple/oathkeeper-api.yaml

This configuration sets up the ORY Oathkeeper API with an in-memory database (please note that restarting the service will remove all existing data!). In addition, ORY Oathkeeper can connect to other database backends such as MySQL or PostgreSQL for persistence.

This configuration also creates a ClusterIP service that makes it available from the Kubernetes-internal network.

But you want the service to be accessible from the outside world! To do that, we’ll fetch the yaml definition

$ wget https://raw.githubusercontent.com/ory/k8s/master/yaml/oathkeeper/simple/oathkeeper-api.yaml

and open it in a text editor. The first section reads the service definition of ORY Oathkeeper:

---
apiVersion: v1
kind: Service
metadata:
  name: ory-oathkeeper
spec:
  ports:
    -
      name: http-ory-oathkeeper
      port: 80
      targetPort: http-api
  selector:
    app: ory-oathkeeper
  type: ClusterIP

This configuration does not include metadata for Ambassador. Let’s change that and make ORY Oathkeeper’s API available to the outside world. In a production deployment, you wouldn’t do this under normal circumstances. Instead you would expose this API internally or with some type of access control in place, such as Ambassador + ORY Oathkeeper!

Ok, let’s define a mapping that makes ORY Oathkeeper available through ambassador. To do so, the metadata of the service needs to be updated:

---
metadata:
  name: ory-oathkeeper
  annotations:
    getambassador.io/config: |-
        ---
        apiVersion: ambassador/v0
        kind:  Mapping
        name:  ory-oathkeeper_mapping
        prefix: /ory-oathkeeper/
        service: ory-oathkeeper

The complete file should now look like this:

---
apiVersion: v1
kind: Service
metadata:
  name: ory-oathkeeper
  annotations:
    getambassador.io/config: |
        ---
        apiVersion: ambassador/v0
        kind:  Mapping
        name:  ory-oathkeeper_mapping
        prefix: /ory-oathkeeper/
        service: ory-oathkeeper
spec:
  ports:
    -
      name: http-ory-oathkeeper
      port: 80
      targetPort: http-api
  selector:
    app: ory-oathkeeper
  type: ClusterIP
[... rest of the file ...]

Let’s re-apply the configuration:

$ kubectl apply -f oathkeeper-api.yaml

Now you can check if the ORY Oathkeeper is alive via the Ambassador route you have created, and you can also list all access rules via the Oathkeeper CLI you downloaded earlier (for now, just an empty array):

$ curl  http://${AMBASSADOR_IP}/ory-oathkeeper/health/alive
{"status":"ok"}$ oathkeeper rules --endpoint  http://${AMBASSADOR_IP}/ory-oathkeeper list
[]

Next, you will define an access rule for accessing ORY Oathkeeper’s API. To keep things simple, you will require no authentication or authorization to access the API. Let’s echo to a new file access-rule-oathkeeper.json:

cat <<EOT > access-rule-oathkeeper.json
[{
  "id": "oathkeeper-access-rule",
  "match": {
    "url": "http://${AMBASSADOR_IP}/ory-oathkeeper/<.*>",
    "methods": ["GET", "POST", "PUT", "DELETE", "OPTIONS", "HEAD"]
  },
  "authenticators": [{ "handler": "anonymous" }],
  "authorizer": { "handler": "allow" },
  "credentials_issuer": { "handler": "noop" }
}]
EOT

You need to make sure that the value match.url (here http://${AMBASSADOR_IP}/ory-oathkeeper/<.*>) has the host and port where Ambassador is available to you. If you set the environment variable previously, this is the case. ${AMBASSADOR_IP}would be, for example, the IP:Port you can find with minikube service list. The rule itself is very simple; it matches all requests with prefix http://${AMBASSADOR_IP}/oathkeeper-api/ and does not enforce any authentication (“anonymous” allows access by unauthorized clients), allows all requests, and does not transform the authorization header. You will set up a more sophisticated rule in the next sections.

Let’s import this rule into ORY Oathkeeper:

$ oathkeeper rules --endpoint  http://${AMBASSADOR_IP}/ory-oathkeeper import access-rule-oathkeeper.json

Now you are ready to activate the external auth service in Ambassador. To do so, you add another section to the annotations you downloaded earlier as file oathkeeper-api.yaml:

---
apiVersion: ambassador/v0
kind:  AuthService
name:  authentication
auth_service: ory-oathkeeper
path_prefix: /judge
allowed_headers:
- Authorization

The complete file should now look like this:

---
apiVersion: v1
kind: Service
metadata:
  annotations:
    getambassador.io/config: |
        ---
        apiVersion: ambassador/v0
        kind:  Mapping
        name:  ory-oathkeeper_mapping
        prefix: /ory-oathkeeper/
        service: ory-oathkeeper
        ---
        apiVersion: ambassador/v0
        kind:  AuthService
        name:  authentication
        auth_service: ory-oathkeeper
        path_prefix: /judge
        allowed_headers:
        - Authorization
  name: ory-oathkeeper
spec:
  ports:
    -
      name: http-ory-oathkeeper
      port: 80
      targetPort: http-api
  selector:
    app: ory-oathkeeper
  type: ClusterIP
[... rest of file …]

And re-apply the configuration:

$ kubectl apply -f oathkeeper-api.yaml

If you retry the command from earlier

$ oathkeeper rules --endpoint  http://${AMBASSADOR_IP}/ory-oathkeeper list
[{
  "authenticators": [{ "handler": "noop" } ],
  [...]

You will notice that the request passes, and you will also see the access rule you just created! Now, if you try to call the httpbin service, the request will fail with a 404 because no access rule has been configured for this service:

$ curl http://${AMBASSADOR_IP}/httpbin/
{"error":{"code":404,"status":"Not Found","request":"84a2b164-7229-4f69-a0cd-227611c07128","message":"Requested url does not match any rules"}}

Let’s change that by creating a simple access rule in file access-rule-httpbin.json for the httpbin service (Don’t forget to replace the URL with your Ambassador IP and port number):

cat <<EOT > access-rule-httpbin.json
[{
  "id": "httpbin-access-rule",
  "match": {
    "url": "http://${AMBASSADOR_IP}/httpbin/<.*>",
    "methods": ["GET"]
  },
  "authenticators": [{ "handler": "anonymous" }],
  "authorizer": { "handler": "deny" },
  "credentials_issuer": { "handler": "noop" }
}]
EOT

The access rule is very similar to the one you created for ORY Oathkeeper. This time however, you are using a simple authorizer that denies all requests. Let’s import the rule and see what happens when you request the httpbin service.

$ oathkeeper rules --endpoint  http://${AMBASSADOR_IP}/ory-oathkeeper import access-rule-httpbin.json$ curl http://${AMBASSADOR_IP}/httpbin/
{"error":{"code":403,"status":"Forbidden","request":"fa893865-35dc-47fd-9907-52da8664c242","message":"Access credentials are not sufficient to access this resource"}}

Ok, so authorization was not granted. Let’s update the rule and allow all requests:

cat <<EOT > access-rule-httpbin.json
[{
  "id": "httpbin-access-rule",
  "match": {
    "url": "http://${AMBASSADOR_IP}/httpbin/<.*>",
    "methods": ["GET"]
  },
  "authenticators": [{ "handler": "anonymous" }],
  "authorizer": { "handler": "allow" },
  "credentials_issuer": { "handler": "noop" }
}]
EOT

Import the file again and execute curl:

$ oathkeeper rules --endpoint  http://${AMBASSADOR_IP}/ory-oathkeeper import access-rule-httpbin.json$ curl http://${AMBASSADOR_IP}/httpbin/
<!DOCTYPE html>
<html lang="en">
[...]

It worked! There are many more authentication and authorization strategies. However, you have barely touched the surface. For example, you can authenticate OAuth 2.0 Access Tokens using the OAuth 2.0 Token Introspection Authenticator. A list of all the possible handlers can be found in the ORY Oathkeeper documentation.

If you’re looking for an OAuth 2.0 Server that works, you should check out ORY Hydra immediately. All ORY products integrate well with one another but can also work completely standalone. The ORY team is also working on an ORY Oathkeeper Authorizer that works with the Open Policy Agent (OPA). If you find this interesting, check out the GitHub issue for this.

Conclusion

In this tutorial, you successfully deployed Ambassador and ORY Oathkeeper to Kubernetes and set up different access rules that grant or deny access to the upstream httpbin service!

Keep an eye out for a follow up blog post that will introduce ORY Hydra and ORY Keto. This will explain how to set up all four services in Kubernetes for a full-stack, cloud native access control system. Sign up for the ORY newsletter to be notified when the blog post is released.

Learn more about Ambassador Labs and the Ambassador Authentication options. If you have any questions, please join our Slack, drop us a line in the comments below, or follow us on @ambassadorlabs Twitter.