Testing Microservices: Isolating Requests (Not Environments) with Telepresence
In the first article of this Testing Microservices series, I discussed the challenges of testing microservices, specifically in relation to the temptation to create large single-developer environments for use for both (inner loop) fast feedback development and (outer loop) integration testing. I presented the concept of testing using developer “request isolation” versus “environment isolation” i.e. testing with a shared environment and isolating the requests traveling through the services here versus testing with your own copy of the entire environment running a copy of all of the services.
The motivation for writing this was to share what I had learned in my consulting experience when creating microservice testing strategies: that reducing the coupling between a developer and their test environments can unblock many testing challenges. The request isolation technique primarily applies to inner loop development and functional testing, but can also help with cross-functional testing, for example, benchmarking or performance testing a locally running single service with realistic inputs.
Microservices can be a useful architectural pattern, particularly to promote autonomy across teams, but there’s no such thing as a free lunch — you have to learn new methodologies for developing effectively with service-oriented architectures. The goal of this second article in the series is to help you build your mental model of request isolation and also to get hands-on with an implementation.
Key ingredients for testing microservices: Request headers, smart routing, and context propagation
The implementation of request isolation starts with making a request to the web page / API / ingress of your system under test and injecting a specific header, say “x-developer-id”. This request header is then propagated from ingress to service to service etc throughout the system running in a shared staging environment. Using this header as a feature flag and identifier at each service (or proxy sitting in front of each service), it is possible to smartly route the request originally destined to a remotely running service elsewhere, for example, to a copy of a specific service under test that is running on your local development machine.
You can implement this “smart routing” with language specific-libraries, hand-rolled proxies, or a CNCF tool like Telepresence.
The key benefit with this approach is that you get to test primarily against the shared staging environment (with all the real dependencies running and accessible there), but with your local copy of the service “swapped in” instead of the corresponding remote service. This enables very fast feedback when testing in the inner and outer development loops.
In the previous article, I mentioned that I often think about this as similar to “copy-on-write” (“copy-on-test”) semantics for the service I’m working on. However, you can also think about this as creating multiple virtual (personal) clusters, each configured with different (isolated) services in the request path and accessed via their own dedicated request/header combination.
Get hands-on: Experimenting to build a mental model for request isolation
I personally like to play with the tech in order to build my understanding of a pattern or technique. Getting hands-on helps me to establish a mental model, and in my experience, the mental models required for cloud native development can be quite different in comparison to more traditional software development methodologies. I’m going to walk you through using the CNCF Telepresence tool in combination with Ambassador Cloud to experiment with request isolation.
The first step I recommend is to take the Ambassador Cloud quickstart for a spin. Here you can get hands-on with the tools without installing anything locally other than a Docker container.
The quickstart deploys an example emojivoto microservices-based application into a remote Kubernetes cluster for you to play around with. Consider this cluster to be your staging environment. Even though it is only you accessing it now, in theory, this could also be where your team is also testing other services.
As you proceed through the quickstart steps you will download and run a Docker container that contains a pre-configured development environment. Within this container you already have the emojivoto source code cloned, the kubeconfig pre-configured to point to your remote K8s cluster, and Telepresence installed. Consider this to be your local dev environment — it’s just that it’s containerized (which is becoming an increasingly popular approach with the likes of Docker Dev Environments and the Microsoft-led devcontainers project, but I digress!)
When you have completed the quickstart you will have generated a secured URL that allows you to view the app via the home page of your remote application, but with your request to the web-app backend service being re-routed to the corresponding service running within your local dev container. You are developing using a hybrid remote-local approach, or “remocal” as Patricia Gaughen and Ana-Maria Cālin from InfluxData called it in a great presentation at the recent KubeCon NA DevX Days event: “From Villains to Heroes: How an Improved DX Has Made Our Devs Happy-ish”.
Under the hood, Telepresence and Ambassador Cloud implement request isolation by injecting a lightweight sidecar (Envoy) proxy next to your remote service which examines every request header and appropriately reroutes any requests that contain an “x-telepresence-intercept-id” header, or simply passes the request through if no header is found. The secured development URL you are using automatically (and transparently) injects this header into the request that is made to your remote application (and also performs some security verification, checking, for example, that only logged-in users can access the application).
There’s no magic involved, but you do get to act like a magician when using Telepresence!
It hopefully isn’t a big mental leap now to see how this concept can be extended. Although it is only you accessing the secured development URL in the quickstart, you could easily share this URL with other developers or stakeholders on your team. Other developers could also spin up a local development container and connect this to the same remote K8s cluster. They could then generate their own secured development URL which is completely isolated (at the request level) from both your secured dev URL and anyone accessing the app normally.
In the final part of this series, you’ll get to explore setting up Telepresence locally and configuring your own services and cluster to take advantage of the request isolation pattern. However, before you do this, I did want to call out several challenges you’ll need to be aware of when adopting this pattern (there’s no such thing as a free lunch in the world of testing microservices!)
Hidden challenges when testing microservices using request isolation
Tools like Telepresence and Ambassador Cloud make testing using request isolation very easy, but there are still a few bigger-picture challenges to watch out for. I’ve been using this testing pattern for many years, and so have made many of the mistakes that I’m about to share here. Hopefully, you can learn from these and make new mistakes instead!
External state mutation: Collisions in shared databases
Even though you can isolate your request and the state contained within your local copy of the service under test, you have to watch for operations that mutate external state, such as data within a datastore. Reading external data is okay, but writing is not. For example, if your local copy of the service writes to an external database (such as MySQL or cloud DB-aaS), this write will be visible to all other copies of the service running in the remote cluster and on other developer’s local machines that are working on the same service with request isolation configured.
An easy(ish) fix for this is to run any local copy of the service under test in a “no-write” profile. Most modern application and microservice frameworks support the notion of profiles, allowing multiple configurations to be defined and switched at run time. For example, a “production” profile will allow the app to write to databases, but a “no-write” profile switches all write operations against the database to be either a “no-op” that does not write any data or scopes any writes within an ephemeral transaction that is rolled back upon completion of the test.
Of course, you will have to design your component tests accordingly, and not rely on reading your writes using this technique. You can (and probably should) still do read/write tests within your CD pipelines.
Accessing external APIs: Collisions and security issues
This is similar to the above gotcha, in that your local copy of the service under test can easily perform a mutating or destructive operation against an external service or API. For example, deleting a user’s credentials in an external identity provider. Even if this is just test data, this can be very annoying for other developers that depend on this data.
Accessing external APIs from a different location (e.g. not your shared staging environment) can also trigger security incidents. For example, many third-party services require any IP address connecting to them to be “allow-listed”. If your machine, running with a different IP than your staging environment, makes repeated attempts to access the API (often getting stuck in a retry loop!), this can trigger a response for a suspected DoS attack.
This issue can be fixed by using the profile approach mentioned above, or by using mocks or service virtualization to return pre-canned responses to requests when testing.
Next steps
Creating an effective microservice testing strategy isn’t easy, but building out a toolbox of both tools and techniques, and understanding when (and when not) to use them, is vitally important.
In my experience, getting hands-on with the technology is always useful for building your mental model of the underlying patterns, usage, and constraints. By using the Ambassador Cloud quickstart you can easily experiment with implementing request isolation using Telepresence without installing anything locally other than a Docker container. Claim a free K8s cluster and take the tour today!
In the next article in this series, we will discuss how to implement this testing pattern using your own services and environments. To be notified when the next part is available, subscribe to our blog on Medium or follow us on Twitter @ambassadorlabs.
Check out the full Testing Microservices series:
- Part 1: You’re Thinking About (Environment) Isolation All Wrong
- Part 2: Isolating Requests (Not Environments) with Telepresence
- Part 3: How to Share Staging Environments without Tripping Over Each Other
