Distributed Tracing with Java “MicroDonuts”, Kubernetes and the Ambassador API Gateway
Distributing tracing is increasingly seen as an essential component for observing microservice-based applications, and many of the modern microservice language frameworks are being provided with support for tracing implementations such as Open Zipkin, Jaeger, OpenCensus, and LightStep xPM. Google were one of the first organisations to talk about their use of distributed tracing in a 2010 paper that described their Dapperimplementation, and one of the requirements that they concluded essential was the need for ubiquitous deployment of the tracing system:
Ubiquity is important since the usefulness of a tracing infrastructure can be severely impacted if even small parts of the system are not being monitored
As I’ve written about previously, many engineers beginning a greenfield project, or exploring a migration, based on the microservice architecture often start by deploying a front proxy or API gateway at the edge in order to dynamically route traffic to independent services. As every inbound request flows through this component, naturally an edge gateway will need to support distributed tracing, ideally using a well-established open protocol.
This article explores how you can add the distributed tracing support provided by the open source Ambassador API Gateway to the existing OpenTracing Java “MicroDonuts” demonstration application running in Kubernetes.
Start with Why?
As discussed by Cindy Sridharan in her very informative “Monitoring in the Time of Cloud Native” blog, not only is distributed tracing considered one of the three pillars of modern observability (alongside metric monitoring and logging), but it provides developers with richer options for debugging distributed systems:
Tracing captures the lifetime of requests as they flow through the various components of a distributed system. The support for enriching the context that’s being propagated with additional key value pairs makes it possible to encode application specific metadata in the trace, which might give developers more debugging power.
In my experience with building and working microservices, tracing has been very useful when diagnosing issues, both in development and production. Understanding the behaviour of a service-based application is often a non-trivial task, and when you combine this with non deterministic behaviour exhibited by the system (particularly if deploying into cloud environments) or communication with unreliable third-parties, the challenge only deepens. Crucially, distributed tracing can give you a window into how a request is handled within your application.
Distributed Tracing 101
The basic idea behind distributed tracing is relatively straightforward — specific inflexion points that a request travels through must be identified within a system and instrumented. These inflexion points include, for example, the API Gateway, each internal service, and data stores or stateful external services. All of the trace data must be coordinated and collated to provide a meaningflow view of a request; this is why you hear about the use of correlation identifiers to enable related trace data to be grouped together for more meaningful analysis:
The CNCF-hosted OpenTracing API is becoming the de facto open tracing standard, and there are several popular open source frameworks that implement this, such as OpenZipkin and Jaeger, as well as commercial options such as LightStep. Many microservice frameworks now offer integrated or compatible tracing implementations. The Java Spring Boot stack provides Spring Cloud Sleuth with Zipkin integration, and the Golang microframework provides OpenTracing wrappers.
Tracing with the Ambassador API Gateway
Recently the Kubernetes-native Ambassador API gateway added distributed tracing support, which is based on the functionality provided by the underlying Envoy Proxy at its core. Ambassador can now generate a request (correlation) identifier and populate the x-request-id HTTP header. Upstream services can forward this header in order to propagate the request context for use in tracing and unified aggregate logging.
The Ambassador tracing implementation currently supports Open Zipkin and Zipkin-compatible backends, such as Jaeger, and the commercial xPM offering from Lightstep. As with Envoy, when using the Zipkin tracer Ambassador adds the B3 HTTP headers, and when using the LightStep tracer the x-ot-span-context HTTP header will be added to any request sent upstream.
Exploring the MicroDonuts Application
The OpenTracing community have very helpfully contributed a series of example applications that demonstrate distributed tracing using all of the implementations mentioned. The “java-opentracing-walkthrough” GitHub repository provides a “MicroDonuts” example that provides traces for a web-based donut ordering application. This example is designed to be run via Maven, and does not need any additional infrastructure, such as Docker and Kubernetes.
The MicroDonuts application is executed as a standalone application (with a single static void main entry point), but it provides several servlets that simulate running multiple services that can be used in the preparation of our donut orders:
void registerServlets() {
kitchenConsumer = new KitchenConsumer();
addServlet(new ServletHolder(new OrderServlet(kitchenConsumer)), "/order");
addServlet(new ServletHolder(new StatusServlet(kitchenConsumer)), "/status");
addServlet(new ServletHolder(new ConfigServlet(config)), "/config.js");
}The tracing component within the application is implemented using the OpenTracing Java SDK and the ”Brave” Zipkin Java SDK. Depending on the configuration file specified (more on this below), the App class initialises the Tracing framework:
} else if ("zipkin".equals(tracerName)){
OkHttpSender sender = OkHttpSender.create(
"http://" +
config.getProperty("zipkin.reporter_host") + ":" +
config.getProperty("zipkin.reporter_port") + "/api/v1/spans");
Reporter<Span> reporter = AsyncReporter.builder(sender).build();
tracer = BraveTracer.create(Tracing.newBuilder()
.localServiceName(componentName)
.spanReporter(reporter)
.build());
}Spans are then created for each innovation of a servlet via an HTTP request. If you look in the ApiContextHandler class you will see the modification I have made to the OrderServlet in order to extract the tracing headers from the current downstream request (which will be made by Ambassador after you make a request to the app via your web browser), and assigns this span as a parent of the new span you are creating for each order of donuts:
@Override
public void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
TextMap headersTextMap = new TextMapExtractAdapter(getHeadersInfo(request));
SpanContext parentSpanCtx = GlobalTracer.get().extract(Format.Builtin.HTTP_HEADERS, headersTextMap);try (Scope orderSpanScope = GlobalTracer.get()
.buildSpan("order_span")
.asChildOf(parentSpanCtx)
.startActive(true)) {request.setAttribute("span", orderSpanScope.span());
...
With these modifications complete, all I needed to do was to package up the application for deployment on to Kubernetes.
Deploying Microdonuts on Kubernetes with Ambassador
Having used the MicroDonuts example to demonstrate tracing concepts at meetups, I often get asked about packaging this application for deployment on to Kubernetes (as this is a popular platform for many organisations to run applications in production). I was keen to test the new Ambassador distributed tracing functionality, and so this provided the perfect excuse to package the MicroDonuts application in Docker and deploy this alongside Ambassador on Kubernetes.
I’ve provided a deep-dive into the approach I took in another post (along with the trials and tribulations I encountered!), but this article focuses on the results, and the aim is for you to get up and running with example within 10 minutes.
First clone my forked version of the project from https://github.com/danielbryantuk/java-opentracing-walkthrough and navigate into the directory
$ git clone https://github.com/danielbryantuk/java-opentracing-walkthrough
$ cd java-opentracing-walkthroughYou will need an empty Kubernetes cluster configured and ready to go. I typically use Google’s Kubernetes Engine (GKE) with ephemeral instances which is configured via the gcloud SDK, as this provides the real cluster experience at a reasonable price point. However, you should be able to use minikube or Docker for Mac/Windows with minimal changes.
$ gcloud container clusters create ambassador-tracing-demo --preemptible
...
kubeconfig entry generated for ambassador-tracing-demo.
NAME LOCATION MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUS
ambassador-tracing-demo us-central1-a 1.9.7-gke.5 35.226.58.170 n1-standard-1 1.9.7-gke.5 3 RUNNING
$
$ # As all GKE clusters enable RBAC by default, create a cluster-admin clusterrolebinding
$ # for your user account
$ kubectl create clusterrolebinding cluster-admin-binding-new \
--clusterrole cluster-admin --user my.user.account@gmail.comFeel free to explore the Dockerfile within this directory, although it is a fairly standard Java image with an OpenJDK 8 JRE. Next, navigate to the kubernetes-ambassador directory which contains of the config files necessary to bootstrap the demonstration:
$ cd kubernetes-ambassador/
(master) kubernetes-ambassador $ ls -lsa
total 304
0 drwxr-xr-x 8 danielbryant staff 256 12 Aug 15:13 .
0 drwxr-xr-x 11 danielbryant staff 352 8 Aug 16:42 ..
8 -rw-r--r--@ 1 danielbryant staff 2043 1 Aug 16:26 ambassador-rbac.yaml
8 -rw-r--r-- 1 danielbryant staff 374 8 Aug 10:43 ambassador-service.yaml
8 -rw-r--r-- 1 danielbryant staff 1145 12 Aug 15:10 microdonut.yaml
8 -rw-r--r-- 1 danielbryant staff 576 12 Aug 14:53 tracing-config.yaml
8 -rw-r--r-- 1 danielbryant staff 1037 8 Aug 11:16 zipkin.yamlThe Ambassador API gateway Deployment and admin Service is configured in ambassador-rbac.yaml file, and a simple rewrite Mapping Ambassador annotation example for the external httpbin.org service is contained in the ambassador-service.yaml file.
Deploying Zipkin and Ambassador in Kubernetes
The Zipkin Deployment and Service is configured within the zipkin.yaml file, which uses the OpenZipkin Docker image. There are two Ambassador annotations in this file: one TracingService to specify the service responsible for collecting Zipkin trace data, and one Mapping that will allow you to navigate to the Zipkin UI and examine traces in your browser. An excerpt of this config is shown below:
---
apiVersion: v1
kind: Service
metadata:
name: zipkin
annotations:
getambassador.io/config: |
---
apiVersion: ambassador/v0
kind: TracingService
name: /tracing/
service: zipkin:9411
driver: zipkin
---
apiVersion: ambassador/v0
kind: Mapping
name: zipkin_mapping
prefix: /zipkin/
rewrite: ""
service: zipkin:9411The MicroDonut application that is defined in microdonut.yaml consists of a single Service and an associated Deployment that uses the container I have added to my DockerHub repository at danielbryantuk/microdonut:1.3. I have also added an Ambassador annotation for the Mapping of the service so that you can order some Donuts via the UI.
If you examine the microdonut.yaml file you will see that I have specified a volume mount for the microdonut container, and backed this with a configMap. The ConfigMap is defined in the tracing-config.yaml file, and the content is used to configure all of the tracing options for the MicroDonut app. This is the file to edit if you want to change from Zipkin to Jaeger tracing, or alter the Zipkin collector Service host or port.
---kind: ConfigMap
apiVersion: v1
metadata:
name: tracing-config
data:
tracer_config.properties: |
public_directory=../client// Selector for the below config blocks
tracer=zipkin// Jaeger config
jaeger.reporter_host=localhost
jaeger.reporter_port=5775// Zipkin config
zipkin.reporter_host=zipkin
zipkin.reporter_port=9411// LightStep config
lightstep.collector_host=collector.lightstep.com
lightstep.collector_port=80
lightstep.access_token={your_token}
You can deploy all of the services and config specified within the YAML files within the kubernetes-ambassador directory like so:
$ kubectl apply -f .
service "ambassador-admin" created
clusterrole "ambassador" created
serviceaccount "ambassador" created
clusterrolebinding "ambassador" created
deployment "ambassador" created
service "ambassador" created
service "microdonut" created
deployment "microdonut" created
configmap "tracing-config" created
service "zipkin" created
deployment "zipkin" createdYou can now query for all of the services via kubectl, although if using GKE you may have to wait a short while before the Ambassador LoadBalancer Service gets an external IP (initially a query may result in “<pending>”
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ambassador LoadBalancer 10.51.248.134 35.224.129.220 80:30306/TCP 1m
ambassador-admin NodePort 10.51.245.207 <none> 8877:32035/TCP 1m
kubernetes ClusterIP 10.51.240.1 <none> 443/TCP 1m
microdonut ClusterIP 10.51.245.91 <none> 10001/TCP 1m
zipkin NodePort 10.51.244.91 <none> 9411:31899/TCP 1mYou can now view the MicroDonut web page by visiting:
http://<external-ip>/microdonut/
You can also view the Zipkin dashboard by visiting:
http://<external-ip>/zipkin/
With everything set up, now all you need to do is order some don
Tracing the Donuts
Now, in order to see some interesting traces, you will need to order some donuts. To do this, simply click on several pictures of the donuts on the MicroDonut web page, and click “order” in the icon that appears below the donut images. After you do this, you should see a countdown in the icon, and the display of a series of steps in the donut preparation phases (“add”, “wait”, “cooking” etc). Feel free to place several orders, although try not to get too hungry when doing this!
Next, open the Zipkin dashboard and click “Find Traces”. Every request that passes through Ambassador (including any request to the Zipkin dashboard) is traced, and therefore you will have to identify a trace involved in the preparation of donuts.
Typically, most of the shorter traces with 2 spans (two service hops) are related to the Zipkin dashboard, and the longer traces with 9+ hops (multiple service hops, and multiple spans created per service) are related to the MicroDonut application. You may need to make several donut orders, and also quickly switch to the Zipkin dash in order to find a related trace, as these can quickly get replaced in the UI search results by other requests being traced by Ambassador.
In the screenshot below, you can see in my example, a donut order created 11 spans and took 184.084 ms to complete.
You can click on the trace in order to get a more detailed breakdown of how the request was handled, e.g.:
Here you can see that Ambassador dealt with the ingress request before passing this upstream to the microdonut service where the “order_span” begins. The “ambassador-default” service name is clearly visible in the trace, and I’m not sure why the “microdonut” service name does not appear, as it does look to be correctly specified within the Zipkin (Brave / OpenTracing) configuration the MicroDonut application.
Another interesting thing to note, is that even though you are making multiple requests to the same MicroDonut application during the ordering and cooking of your donuts, each request is an out-of-process HTTP request made via the localhost loopback adapter (and so not going via Ambassador), and the span information is being propagated by the Brave implement in order to join all of these traces up correctly.
What’s Next?
Given this sample application and Ambassador config, you should be able to get any Zipkin compatible application tracing up and running with minimal hassle. As Ambassador injects the Zipkin headers into any upstream request, any application that recognises these headers (and propagates them onwards) should be traceable.
The example Java code shows how to implement the Zipkin header processing using the Brave library (and also how to attach the Ambassador generated span as a parent to each child span), and I’ll go into more detail of how I modified the MicroDonut example in other post. However, don’t let the fact that the example was written in Java stop you. Any language or framework that supports Zipkin should work right out of the box with the provided config — all you need to specify is the Zipkin Kubernetes Service host and port, and you should be good to go!
You can learn more about Ambassador at https://www.getambassador.io, and about the Ambassador Distributed Tracing feature in the Ambassador docs. If you have any questions, please join our Slack, drop us a line in the comments below, or @getambassadorio on Twitter.
This article originally appeared on the getambassador.io blog.
