Why IT Ticketing Systems Don’t Work with Microservices

Published in

Ambassador Labs

5 min readOct 24, 2018

To effectively build cloud native microservices applications, your engineering organization must adopt a decentralized decision-making culture to move faster. You will also need the ability to use self-service platforms and tooling to implement operational concerns, such as provisioning infrastructure, declaring resource requirements for services, and implementing (and alerting based upon) monitoring. If you want your teams to move independently and at speed, and avoid potential bottlenecks, then it is not practical to raise tickets in a support system to accomplish these tasks.

Cloud Native Patterns

In this article series, we discuss key patterns in cloud native application development and explore why they’re effective and how to implement them in your organization. We also examine the consequences of implementation (both good and bad) and provide examples using popular cloud native tools.

In the second part of this series, we discuss “soup to nuts self-service”, which is a pattern focused on offering self-service functionality throughout the entire software development and deployment cycle.

Intent

“Soup to nuts self-service” is a practice that enables fast feedback for engineers, from the generation of a hypothesis or an idea for new functionality to a running customer-facing experiment or implementation. Offering self-service provisioning of infrastructure, development environments, and observability tooling also reduces potential bottlenecks and reliance on a central operations team when engineers are working within the inner development loop and increases ownership of “software products” across teams.

Motivation

“Soup to nuts” is an American English idiom that conveys the meaning of “from beginning to end”, and is derived from the description of a (perhaps old-fashioned) full-course dinner, in which courses progress from soup to a dessert of nuts.

The phrase “soup to nuts self-service” was chosen to convey the end-to-end nature of self-service required within the typical software development lifecycle. With the rise in popularity of DevOps and other related methodologies, many organizations appreciate the need to minimize hand-offs and prevent software artifacts from being “thrown over the wall” from development to operation teams.

However, fewer teams have recognized the need to be fully self-sufficient, to be capable of taking a business hypothesis, prototyping code, experimenting by exposing a fraction of real user traffic to this code, and evaluating success against observable metrics and key performance indicators (KPIs). The engineering teams that can address user needs and iterate the fastest will typically gain the biggest market share in a competitive market.

The genesis of this pattern can be seen within Deming’s Plan-Do-Check-Act (PDCA) cycle or Boyd’s observe-orient-decide-act (OODA) loop.

In modern software development, we believe that the OODA loop (per product) is realized as a four-phase cycle: develop, test, release, and observe. The diagram below of the cycle purposely does not line up our phases with the OODA loop diagram phases, as they aren’t directly analogous:

Applicability

Use soup to nuts self-service when:

An application consists of multiple microservices that are changing at independent rates, and relying on a centralized (ticket-driven) operations team would become a bottleneck to getting rapid feedback
The company is (or is planning to become) organized around product teams, and these teams will be responsible for the entire lifecycle of an application
Expertise within an organization is divided into full cycle (full stack) developers and platform engineers

Do not use this pattern when:

The engineering teams are not operationally aware
The engineering teams have no understanding of the business goals
There are regulatory or compliance issues which enforce multiple stages of hand-offs and verification (although this is becoming less of a barrier, as demonstrated by many talks at the DevOps Enterprise Summit)

Structure/Implementation

The soup to nuts self service pattern is typically implemented across a range of technologies with differing utility. For example, at one end of the spectrum, the development team is simply given access to the bare metal servers, VMs or cloud APIs/SDKs and can do whatever they require. This implementation style requires that the development teams are very operationally skilled and can also be trusted to be enabled to deploy and change anything they require.

At the opposite end of the spectrum, the development teams are given tightly-scoped access to a PaaS or FaaS platform, where they request and configure infrastructure by declarative configuration files and push and verify proposed updates via version control (e.g. ‘cf push’ in Cloud Foundry). This implementation style only requires that development teams understand the workflow and properties of the platform, and safeguards or guide rails can be added at an administration level and enforced within build pipelines.

Here are some implementation issues to consider:

A prerequisite to implementing this platform is that the underlying platform can be manipulated via a programmatic API or SDK
The developer control plane can be implemented via CLI and/or UI
Unless the platform is provided by a third-party, a dedicated platform team will be required in-house

Consequences

Using the soup to nuts self-service pattern has the following benefits:

Engineering teams become independent, loosely coupled and can rapidly deliver experiments and functionality to end users
Learning from experimentation in a production environment (with minimum risk), encourages the development of both empathy with the end-user and mechanical sympathy with the underlying technology platform
Outcome-oriented engineering teams can organize around products, and effectively become software product owners
Total IT expenditure can become easier to plan, with outcome-oriented teams who deliver value to end users, and output-oriented teams who deliver some value meant only for consumption within the organizations wall

And liabilities:

Initial upfront investment in establishing a platform and associated tooling to enable self-service tooling can be high. There is also a danger of “not invented here (NIH)” issues, where engineers are reluctant to use (or buy) existing solutions and instead choose to write their own
With less oversight and guide rails, development teams can release functionality that negatively impacts the end user, either directly through functional requirements (e.g., providing a bad user experience or limiting accessibility requirements) or indirectly through system quality attributes (reducing performance or leaking credentials from the application)

Example

Public cloud vendors APIs, SDKs and portals e.g., AWS, GCP, Azure etc
First generation VM management tooling e.g., VMware vSphere, OpenStack
IaC tooling e.g., Terraform, Chef, Ansible, Puppet, SaltStack etc
Orchestration/PaaS e.g., Kubernetes, Mesos, CloudFoundry (declarative configuration)
Communication e.g., Ambassador API gateway, Istio, Consul Connect
Inner development loop tooling e.g., Draft, Skaffold, Ksonnet etc

Known Uses

Related Patterns

API gateway
Canary releases
Local-to-remote debugging