APPLICATION MODERNIZATION

Part 4: Incremental App Migration from VMs to Kubernetes — Implementing End-to-End TLS

Using Ambassador API gateway and Consul service mesh to implement user-to-service security

Published in

Ambassador Labs

6 min readSep 4, 2019

If you have been following along with previous articles in this series, you will now have a template that you can use to spin up a Kubernetes cluster with an API gateway, and you can deploy (and migrate) containerized apps here, and also route to services running outside the cluster on any platform, such as bare metal, VMs, or serverless. This provides you with the ability to incrementally migrate from your existing platform to Kubernetes, or run a hybrid platform configuration. Another key ability that you require is being able to securely migrate between platforms, and that is what this article focuses on.

Both the technology and mainstream press are increasingly filled with stories of IT breaches and hacks, and although many of the reported attack vectors are quite simple, there is nearly always a chain of failures that led to the breach. Implementing a defense in depth strategy is key here, as the more difficult you make a hack, the more chance that a hacker will be defeated, get caught, or switch targets.

I’ve recently presented a series of conference talks and webinars, alongside Nic Jackson, that focus on this topic.

Due to the scope and focus of this article, I’m not going to cover essential security topics like hardening infrastructure, scanning code artifacts for vulnerabilities, or encrypting data at rest. I will, however, focus on encrypting data in transit and service-level authorization.

End-to-end TLS with Ambassador and Consul

In part 3 of this series you deployed the Ambassador API gateway and Consul service mesh to route a request from an end user through to a service. Here you were simply using Consul as a service discovery mechanism for Ambassador to match an API endpoint to a backend service. But Consul can do much more than this, especially when it is configured to act as a service mesh.

Adding “north-south” TLS termination to Ambassador, whether a request is handled via a downstream CDN or directly by Ambassador itself, is simple, and the docs should provide everything you need (I also recommend integrating JetStack’s cert-manager project with Ambassador, as this automatically manages the TLS certificate renewal process for you).

Adding “east-west” TLS support for service-to-service traffic is just as easy with Consul, although you will need to configure an Envoy Proxy sidecar to run next to each service. Let’s explore this in a little more detail.

Implementing TLS communication for Kubernetes Services

Thanks to the Consul Helm chart and the consul-k8s project, it is super-easy to add a Consul-managed Envoy sidecar to your existing or new Kubernetes services. If we look at the api.yml configuration file from Nic’s emojify-app we can see four lines of annotation configuration on the Deployment to automatically inject and configure the sidecar, and only one of these lines is mandatory: “consul.hashicorp.com/connect-inject”: “true”:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: emojify-api
  labels:
    app: emojify-api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: emojify-api
  template:
    metadata:
      labels:
        app: emojify-api
      annotations:
        "consul.hashicorp.com/connect-inject": "true"
        "consul.hashicorp.com/connect-service-protocol": "http"
        "consul.hashicorp.com/connect-service-upstreams": "emojify-facedetect:8003,emojify-cache:8005"
        "prometheus_io_scrape": "true"

If you annotate all of your Kubernetes services in the same way, and communicate via the service names or ports configured in the Consul configuration, all of your service-to-service traffic will be sent via an encrypted channel.

Adding encrypted communication from Ambassador into the mesh is simple, too. The Ambassador and Consul Integration docs provide complete instructions, but in essence all you have to do is deploy the ambassador-consul-connector.yaml config into your cluster (that defines RBAC resource, connector service, and TLSContexts),

$ kubectl apply -f https://www.getambassador.io/yaml/consul/ambassador-consul-connector.yaml

and add the tls property that points to the ambassador-consul TLS configuration on your existing Mappings e.g.:

---
apiVersion: getambassador.io/v1
kind: Mapping
metadata:
  name: consul-api-mapping
  namespace: default
spec:
  prefix: /api/
  timeout_ms: 20000
  host: emojify.today
  service: emojify-api-sidecar-proxy
  resolver: consul-dc1
  tls: ambassador-consul
  load_balancer:
    policy: round_robin

It really is as easy as this for Kubernetes-managed services. Things get slightly more complicated when running an out-of-cluster service as you have to add the external node to the Consul cluster and also run the Envoy sidecar yourself, but there are several tools to help you do this. Let’s explore this more now.

Configuring TLS communication for out-of-cluster services

In part 3 of this series I discussed the need for configuring a routable path from Ambassador to any out-of-cluster node, and also discussed several options for doing this. In my demonstration application I’m deploying onto Google Cloud Platform (GCP), and this platform provides “IP Aliasing” for Kubernetes Pods, which allows any compute nodes deployed into the same Virtual Private Cloud (VPC) to communicate with a Pod via it’s cluster IP. Both Azure and AWS also offering similar functionality, although I haven’t personally tested this.

I’ve found that the easiest method of spinning up an Envoy Proxy sidecar on a non-Kubernetes node, is via Docker. For a production use case you would want to run this via something like systemd, or you could also install the Envoy binary from other sources (or build you own copy) and run this process via systemd.

Providing you have deployed the sample GCP playground from part 3, then you can ssh into the shopfront instance and install Docker

$ sudo apt install -y docker.io
$ # add the current user to the docker group, to remove the need to sudo
$ sudo usermod -aG docker $USER

As specified in the Consul documentation, you can then build a custom Docker image with the Consul and Envoy binaries:

$ cat  > Dockerfile <<EOF
FROM consul:1.5.2
FROM envoyproxy/envoy:v1.8.0
COPY --from=0 /bin/consul /bin/consul
ENTRYPOINT ["dumb-init", "consul", "connect", "envoy"]
EOF

With this complete, you can now build the container:

$ docker build -t consul-envoy .

Before running the Envoy side car, you will want to update your Consul configuration for the shopfront service:

{
   "service":{
      "name":"shopfront",
      "tags":[
         "springboot"
      ],
      "address":"10.128.0.5",
      "port":80,
      "connect" : {
      "sidecar_service" : {}
      }
   }
}

And now, all you have to do is run the Consul-Envoy Docker image. Be sure to specify the -sidecar-for shopfront configuration correctly:

$ docker run --rm -d --network host  --name shopfront-sidecar-proxy 
  consul-envoy -sidecar-for shopfront -- -l debug

Give this node a few seconds to register with the Kubernetes-based Consul cluster, and you will then be able to make a request for the shopfront service via the k8s-hosted Ambassador API gateway, which will route traffic via TLS to the external instance:

Conclusion and looking to the future

Moving services to new infrastructure or running hybrid platforms is becoming increasingly common with application modernization programs. A core requirement of any such program is that you must be able to conduct the migration both incrementally and securely. This article series has provided a reference architecture and approach for doing this using Kubernetes, the Ambassador API gateway, and the Consul service mesh.

The combination of Ambassador and Consul makes it very easy to implement traffic encryption with TLS across the entire cluster. Ambassador also integrates well with popular CDNs like Cloudflare or Akamai, if you want to terminate end-user TLS here, and the HashiCorp team behind Consul have recently released a new mesh gateway feature that will allow encrypted traffic to travel between multiple clusters. As with any approach to implementing security, you do have to “mind the gap”, but with appropriate tooling and testing, this is completely doable with modern cloud native technologies.

This article series and the associated codebase will continually evolve, and so please reach out to me if you have any particular requests for cloud vendors or complicated routing scenarios.

As usual, you can also ask any questions you may have via Twitter @ambassadorlabs, Slack or raise issues via GitHub.