Extending the Ambassador/Java Distributed Tracing with LightStep

Daniel Bryant

·

Follow

Published in

Ambassador Labs

·

7 min read

·

Oct 3, 2018

--

Listen

Share

Last month I published a tutorial on Zipkin-based distributed tracing using the Kubernetes-native Ambassador API gateway and the Java OpenTracing ‘Microdonuts’ application. After a few requests I wanted to loop back around now and add LightStep support to this.

Introducing LightStep [x]PM

For those new to LightStep, it is a SaaS-based distributed tracing platform (and more) that aims to provide the next generation of Application Performance Monitoring (APM) — [x]PM. If you’re interested, I wrote about some of the history in the distributed tracing and APM spaces on InfoQ recently. The satellite-based architecture of the LightStep platform is interesting, as is the UI and diagnostic tooling provided, and I encourage the interested reader to check out their comprehensive docs.

In essence, the LightStep [x]PM data processing pipeline has two major components: the Satellites, which are standalone software appliances running within a customer’s network or VPC, and the LightStep Engine, the SaaS component. All spans generated by instrumented clients and servers are sent to a pool of satellites, where they are processed and temporarily stored during trace assembly. The LightStep Engine queries the satellites, records aggregate information about spans, directs the assembly process, and stores traces durably.

You can experiment with the SaaS platform via the LightStep website, and you will need to create an account and get an access token in order to programmatically send span data to LightStep.

Configuring the OpenTracing Java ‘Microdonuts’ Application

If you clone my fork of the java-opentracing-walkthrough GitHub repo and list the directory contents, you should see something similar to this:

$ git clone git@github.com:danielbryantuk/java-opentracing-walkthrough.git
...
$ cd java-opentracing-walkthrough
$ ls
Dockerfile                      README.md                       kubernetes-ambassador           microdonuts
LICENSE                         client                          kubernetes-ambassador-lightstep

Most of the content is the same as my original tutorial, and you can read this if you want more details. The primary additions for the LightStep integration are located within the Kubernetes-ambassador-lightstep folder. If you navigate to here and list the contents, you should see something similar to this:

cd kubernetes-ambassador-lightstep/
$ ls
ambassador-rbac.yaml    ambassador-service.yaml lightstep-config.yaml   lightstep-tracing.yaml  microdonut.yaml         tracing-config.yaml

The LightStep tracing config is located within the lightstep-tracing.yaml file. If you open this YAML file within your favorite editor, this is what you should see:

---
apiVersion: v1
kind: Service
metadata:
  name: lightstep
  annotations:
    getambassador.io/config: |
      ---
      apiVersion: ambassador/v0
      kind: TracingService
      name: tracing
      service: "my-collector-grpc.mydomain.com:443"
      driver: lightstep
      config: {
          access_token_file: /config/lightstep_api_key.txt
      }
spec:
  type: ExternalName
  externalName: my-collector-grpc.mydomain.com

You’ll recognize the Ambassador annotation and most of the TracingService details from the previous tutorial, but you’ll also notice that in addition to changing the driver from zipkin to lightstep, I have also defined the TracingService to report the traces to an external URI (my-collector-grpc.mydomain.com, which is where my LightStep trace collector is running outside of the Kubernetes cluster) and specified an access_token_file property within the TracingService config.

The observant among you will have most likely been pondering, “how do we get the /config/lightstep_api_key.txt file into Ambassador? Great question. Ambassador simply runs as a container within a pod, which is in turn defined within a Deployment, just like any other Kubernetes service. Therefore you can mount a ConfigMap as a volume into the Ambassador container with the relevant data.

Let’s first define a ConfigMap, an example of which you can find in the lightstep-config.yaml file:

---kind: ConfigMap
apiVersion: v1
metadata:
    name: lightstep-config
data:
    lightstep_api_key.txt: <your key here>

If you are following along, and have a LightStep account, then replace <your key here> with your actual key text. With the ConfigMap defined, you can now map this into the Ambassador container. If you look back into the ambassador-rbac.yaml file, you can see at the bottom of the config the volumeMounts and volumes defined (note the … shows where I have truncated the config to improve readability here):

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: ambassador
spec:
  replicas: 3
  template:
    ...
    spec:
      serviceAccountName: ambassador
      ...
        volumeMounts:
        - name: lightstep-config-volume
          mountPath: /config
      restartPolicy: Always
      volumes:
      - name: lightstep-config-volume
        configMap:
          name: lightstep-config

With all of these pieces of the puzzle complete, you can now deploy Ambassador with LightStep Tracing into Kubernetes. I’m going to use Google’s hosted GKE service (with preemptible VMs, just to keep the cost low), but you can use whatever flavor of Kubernetes that works for you.

$ gcloud container clusters create ambassador-tracing-demo --preemptible
...
Creating cluster ambassador-tracing-demo...done.
...$ kubectl create clusterrolebinding cluster-admin-binding-new \
--clusterrole cluster-admin --user < my GCP user name>clusterrolebinding "cluster-admin-binding-new" created$ kubectl apply -f lightstep-config.yaml,ambassador-rbac.yaml
configmap "lightstep-config" created
service "ambassador-admin" created
clusterrole "ambassador" created
serviceaccount "ambassador" created
clusterrolebinding "ambassador" created
deployment "ambassador" created

Let’s now go ahead and deploy the LightStep tracing config:

$ kubectl apply -f lightstep-tracing.yaml 
service "lightstep" created

You can now deploy a simple httpbin Service to the cluster, instructing Ambassador to proxy requests from the /httpbin/ route to the external httpbin.org website.

$ kubectl apply -f ambassador-service.yaml 
service "ambassador" created

Now you can query Kubernetes for the Ambassador Service LoadBalancer external IP address. Here is what I saw:

I can see that Ambassador is exposed on the external IP of 35.239.206.165, and so I can make a request to my httpbin service, querying for the IP by specifying the ‘ip’ endpoint in the httpbin API, like so:

$ curl 35.239.206.165/httpbin/ip
{
  "origin": "146.148.98.46"
}

You can curl your Ambassador httpbin endpoint a few times to make sure you have created a few traces. If you now log in to your LightStep account and navigate to the LiveView, you should see something similar to this:

This view provides a good overview of current traces, both the latency via the histogram in the top half of the page and the additional individual trace information in the lower half of the page. You can click on the individual traces to get additional information on the trace:

Let’s now deploy the MicroDonut application and associate ConfigMap with our configuration tracing. You’ll note now that I have updated the Java application’s tracing config to send its trace data to LightStep, separate from the work we have already done to configure Ambassador. The tracing-config.yaml file now contains this:

---kind: ConfigMap
apiVersion: v1
metadata:
    name: tracing-config
data:
    tracer_config.properties: |
        public_directory=../client        // Selector for the below config blocks
        tracer=lightstep        // Jaeger config
        jaeger.reporter_host=localhost
        jaeger.reporter_port=5775        // Zipkin config
        zipkin.reporter_host=zipkin
        zipkin.reporter_port=9411        // LightStep config
        lightstep.collector_host=< your lightstep collector host >
        // The collector_protocol value is either "http" or "https"
        lightstep.collector_protocol=http
        lightstep.collector_port=8081
        lightstep.access_token=< your access token >

If you are following along, you will need to replace <your access token > with you actual text token.

Let’s deploy this into Kubernetes:

You should now be able to access the MicroDonut app via your web browser via the Ambassador external IP and the route /microdonut/ e.g.:

Before you get too hungry, you can order some virtual donuts by clicking on the images and pressing the “order” button that is displayed below the images. Do this a few times to generate a few traces, and navigate to the LightStep LiveView:

That’s it! Now you can explore and experiment with Ambassador, LightStep, and MicroDonuts.

Conclusion

Learn more about Ambassador Labs and about Ambassador Distributed Tracing feature in the Ambassador docs. Try LightStep today! If you have any questions, please join our Slack, drop us a line in the comments below, or @ambassadorlabs on Twitter.

Acknowledgements (a Big Thanks!)

I would like to thank Alex Gervais from AppDirect for all of his help during this work. Not only did Alex provide guidance, but he has previously done most of the work on the open source Ambassador Tracing contributions and also fixed a recent bug with the integrations. Thanks to the LightStep team, who gave us access to a trial account and also provided useful feedback on the article as it was being written.