Traffic Shadowing and Dark Launch - Embrace the Dark Side of API Gateways
You’ve probably bumped into the phrase “dark launch” when reading about software feature releases by the unicorn companies: Facebook, Google and Amazon etc. In fact, Facebook coined the term when describing the launch of their new chat feature, in which they deployed the code responsible for the new chat and sent requests to this service but did not show the results to end-users. This allowed the engineers to monitor the new chat service using real production traffic without impacting the user experience.
What is Dark Launch?
With dark launch, you can release production-ready software features to a small group of users while hiding them from the rest. You might also use this process for deploying code and exposing it only in an environment without any traffic going through its exposed parts — think beta testing but with no human involved.
Traffic Shadowing & Dark Launch with Ambassador
If you stopped by the Ambassador Labs booth at KubeCon EU, you would have most likely seen our demonstrations with shadowing traffic via the Kubernetes-native Ambassador Edge Stack API Gateway. The ability to shadow, or duplicate, incoming ingress requests from your users allows you to dark launch new services with ease. The Prometheus integration with Ambassador also allows you to monitor top-line metrics such as latency and the number of 5XX errors generated. You can easily aggregate these metrics with additional service metrics, and practically every modern programming language has good integration with Prometheus.
Managing State and Third-Party Interactions
Not all services are easy to test with dark launch. In particular, services that mutate state or interact (in an “unsafe” way) with third-party services require a different strategy than testing a purely stateless service. To illustrate the challenges, consider this thought experiment. Say you want to dark launch a new checkout service as part of an e-commerce application. This service not only interacts with a third-party payment service like Stripe in an “unsafe” way, but it also persists the state of a purchased shopping basket and sends a global event of a completed purchase to any other services listening. It is easy to imagine that for every user whose checkout request follows the dark launch path, they end up with two debits on their credit card and potentially two lots of goods being delivered.
When dark launch this type of service you either need to run the dark service in a different profile that disables interaction with third-parties and data stores or potentially (from the perspective of realism and monitoring potential) replace the third-party service and datastores with a virtualised representation. For example, the lightweight service virtualisation tool Hoverfly can be used to simulate third-party interactions. And for embedded datastore and middleware tools like:
- HyperSQL (in-memory MySQL substitute)
- Apache Qpid (in-memory AMQP)
- LocalStack (AWS service and datastore simulator)
Dark Launch as an Option
Dark launch a feature is just one option in your deployment toolbox, and ideally you would combine the use of this with other approaches like canary testing. For example, after testing the operational profile of a new feature by dark launch it using traffic shadowing, you would then test the user experience perspective by gradually canary releasing a feature to an increasing number of your users over time.
Visit our website to learn more about Ambassador Labs, and about the Ambassador Traffic Shadowing feature in the Ambassador docs. Join Ambassador Labs Community on Slack, or at @ambassadorlabs on Twitter.
