Testing in Production: Canary Releasing with Kubernetes
No code is ever truly proven or tested until it’s running in production. This is particularly true of today’s cloud-native systems; these applications are rapidly evolving, continuously deployed, and have multiple layers of dependencies. As we’ve argued previously, traditional testing strategies such as mocks, staging environments, and integration testing are not sufficient for testing cloud-native applications.
Canary releasing is a technique to reduce the risk of introducing a new software version in production by slowly rolling out the change to a small subset of users and verifying the results. Of course, the ultimate goal with a canary release is to roll out the new software to the entire infrastructure and make it available to everybody. Still, you can limit any potential blast radius of issues and easily roll back using the same mechanism you are controlling the release.
Canary Releasing with Kubernetes: Deployments
The first approach to Kubernetes canary releases that many engineers discover is implemented by creating a Service backed by multiple Deployments with several common selector labels and one “canary” label. For example, the Kubernetes documentation suggests using multiple “track” labels, such as “stable” and “canary”, in addition to the common identifying Service selector labels (e.g. “app: guestbook”). Assuming that a round robin load balancing algorithm is being used within the Service, the percentage of traffic directed to the canary can be selected by altering the ratio of “stable” to “canary” Deployments.
The obvious downside to implementing Kubernetes canary releases like this is that if the application contained within the Deployment is large or resource-intensive it may not be practical to deploy multiple versions of this.
Canary Releasing with k8s and a Smart Edge Proxy
A more effective approach to Kubernetes canary releases is using a smart proxy, load balancer or API gateway — like Ambassador Edge Stack. Instead of relying on a round-robin load balancing implementation, the smart proxy can direct a specified percentage of Service requests to each Deployment.
Dynamically routing ingress traffic at the request level means that only one Deployment is required for each “stable” and “canary” versions of the application, regardless of the percentage of traffic you want to route. If you can expose your metrics via an API (Ambassador reports top-line metrics to Prometheus), then you can automate a large part of the incremental traffic routing and verification.
More on Canary Releases
We’ve seen many of our users using Ambassador in this way, so we’ve started documenting some best practices for canary releases in Kubernetes. You can also learn more about implementing canary releases with Ambassador in the Ambassador docs. To read more, check out our Comprehensive Guide to Canary Releases.
