Determining whether automation is helpful beyond the CI/CD pipeline can be easier through a break-even analysis
The pursuit of automating away all the operational tasks in your organization doesn’t stop after you finish building a solid CI/CD pipeline infrastructure using the myriad tools available and making it stable.
There are many more one-off tasks not part of the CI/CD pipeline, but which are very important for managing the infrastructure and operating the applications running in production, that are not yet automated. These tasks could be very specific to your organization, but some common categories of these tasks are:
- Provisioning and maintenance of infrastructure, including patching, updating certificates and licenses.
- Maintenance of monitoring systems.
- Generation of reports and dashboards.
To Automate or Not to Automate?
A CI/CD pipeline is an end-to-end automated system that handles all aspects, from builds triggered by code change through deploying the built artifacts into production, with very little or no human intervention in between. The non-CI/CD tasks described above could be automated or be carried out manually.
An inexperienced DevOps team can easily get carried away by the notion of automating everything and end up spending time on low-priority tasks—sometimes, simply because using a new tool or the coding work involved in automating a task can be very attractive to the team members. While such endeavors would look great in a sprint demo and on a resumé, they won’t help the business achieve operational velocity. When there are multiple automation projects competing for the team’s time, it would make sense to do a break-even analysis to obtain better insight on the returns of automating an operational task.
The Agile methods to select and quantify a task such as backlog prioritization and estimating time for a story are not adequate to assess how the organization would benefit from that project overall. The break-even analysis proposed here would help to determine whether an automation project is worth doing by looking at the overall impact of rolling out that automation in the organization. Agile is limited by its narrow focus on the team that rolls out the automation—the DevOps team—and it cannot provide any metrics on the overall impact of an automated process on the organization.
The Break-Even Analysis
Let’s define various metrics involved first and then do the simple math of overheads and breaking even with the benefits.
Development time (d) – This is the time needed to develop and roll out an automation task. This would be same as the time estimate used for a related story in an Agile sprint. In hours.
Maintenance time (m) – As any experienced DevOps engineer can attest, developing a script or tool suite is not a one-time deal. The changes will be needed due to many factors:
- Infrastructure change.
- Fixing bugs in the tool.
- Enhancement requests.
- Training users of the tools, such as SREs and fellow team members.
Budget the hours needed annually for this task, depending on the complexity and possibility of changes down the road.
Manual working time ™ – The time needed to complete the task without automation. Doing things manually can cause human errors which adds to the total time. Factor that in the estimate if this task has been causing issues. In hours.
Automated working time (ta) – The time needed to complete the task using automation. In hours.
Jobs count (c) – The number of times this task is run. Make an annual estimation so even infrequent operational tasks can have a score.
Though arbitrary, my suggestion is to try out an automation for a year for its benefits. Check if the project would bring in impressive returns during that time frame:
Project overhead = d + m
Savings from automation = (tm – ta) x c
Net benefit to organization = Savings from automation – Project overhead
To embark on an automation project, the organization should benefit greatly from it to do things faster and better. While an Agile way of prioritizing an automation task would narrow the focus only on the DevOps team, this method also gauges and factors in the users’ time.
DevOps Benefiting Customers
The tools developed might not be used internally always, which is a major assumption in the above break-even analysis. It is quite possible that an automation process could benefit end users as well. I think it is very important to factor in the time saved by our customers also.
Happy customers always bring in goodwill to a company.