DevOps Healing

Evgeny
ProdOpsIO
Published in
8 min readJul 5, 2018

--

A short presentation that I made during the DevOps in Israel meetup event about healing organizations with DevOps culture.

Link to the event —
http://www.meetup.com/devops-in-israel/events/157983792/

This blog post includes slides and a bit of narrative for each of the slides. It is quite long, but the subject seemed important enough to use all these extra words.

The presentation talks about our experience of taking some organizations that start to feel not too well at one point. And using our experience from other organizations that seem to be feeling better, and applying these techniques, methodologies, and tools to heal. Devops culture has a lot of ideas that are used almost as-is to make this happen.

Some traditional organizations are still stuck in the “Ain’t broke” phase. For various reasons this is problematic and usually is the main reason that is causing stagnation and barrier to move faster and improve. We know from Devops that continuous improvement is a pillar for health, this approach should be “banned” or at least shunned upon.

Most technology companies that respect themselves want to evolve and become better, and usually, this is associated with Innovation. Having any kind of approach ingrained in the organization that contributes to stagnation is directly resulting in making Innovation impaired and not enabling to move forward fast enough.

Startups that don’t move fast enough just fail, and to innovate they cannot afford themselves leaving things “unfixed”. Enterprises on the other side of the spectrum have smaller, leaner, faster competitors coming in and are facing the risk of becoming obsolete if they don’t embrace innovation as a way of life.

The book Lean Startup that has been widely popular recently talks, as I see it, about the need for continuous feedback. Without doing small steps and verifying that these worked, it is very hard to innovate. The alternative of the Lean way makes us think of monolithic Waterfall projects where a huge chunk of work would be commissioned at the start, and feedback would be very late in the game. Companies realized that these days when the world is fast, they cannot afford to behave that way anymore.

The new way of doing things, the way startups and businesses in general, today are becoming competitive is doing everything better, spending less time and money and taking very limited risks in doing so. Jumping into the water with lots of investment of time and money is very risky and might lead to a bad system without even knowing it until it is too late.

This is why startup are usually created with small teams that enables them to move fast and get to market without wasting too much time, money, resources and getting validation of their idea/product much faster than before.

Lean startup and other ideas of the same nature are starting to disrupt the workplace. Small teams are often found even in big enterprise companies, and this enables them to remain competitive and get great products to market.

Moving fast is the name of the game.

But when becoming successful, usually what follows is growth. Small fast teams at startups become large companies with more and more and more new people coming onboard every day. This is where usually things start to break and become a little bit unhealthy. The tools and methodologies and processes that used to work for a team of 4 or 5, now have some ill effects that are not even surfaced.

When teams are large, or people work in small teams in large companies, they usually are not as willing to “fix” what is bothering them in their daily work. When something is broken, no one feels that he has the power to fix it when he or she is part of a larger organization. Even when people do complain, they don’t feel they have the power to change … and this is not always true, it might be just in their heads.

bit.ly/vanguard-method

Before talking about which tools we can use to fix things in an organization, and before we talk about what to change — we need to see how to understand the current situation. John Seddon has an amazing presentation from Oredev conference that shows how Systems Thinking can be applied to study the actual work being done, even before any changes are made. This is important. Without learning about what is actually happening, a manager or the CEO might think that they “know” what to change to fix things, in most cases, they are completely wrong.

Links to John’s talk -

In a technological company, one of the main things that enable visibility to the current state is measuring. As the amazing folks at Etsy explain “Measure anything, measure everything”. The above picture is a wall in the Etsy office where a lot of information hangs for everyone to look at and think about, this leads people to try and think about ideas for improvement. This is empowering. This enables to understand the current state like nothing else does. Without measuring everything and anything, it would be very hard to come to conclusions about how things actually work.

What we have recently done together with several of our customers are interviews. Talking to team leaders, and generally, people in the field can give a lot of insight about how processes are actually being executed — it usually reveals “dark IT” and other hidden problems that management might not even be aware of. Letting people speak without putting too many words in their mouth is a rare sight to behold, apparently, this is not done in most organizations — and just coming to people and talking to them provides a stage for all the day-to-day problems to become recognized and “documented”.

Write it down. Whatever people are saying, write it down, or record it in some way — so that a summary can be made without omitting all the details and testimonies. These interviews provide pure gold, and it is a real shame when it is not recorded and instead gets lost completely and forgotten. That is not the reason why these are made, so in order to not forget, write stuff down.

When measuring and interviewing, getting the actual picture of what is happening — it is important to take note of certain important topics. The more traditional “Lean” methodology, mixed with “Toyota Production System”, “Six Sigma”, and all the other great things that came from that side of manufacturing methodologies — these all talk about waste. Briefly, waste has different types — but it all boils down to things that increase costs and time spent, or in other words, the investment made. And on the other hand, things that do not contribute much value, or even reduce the value that the whole system (remember, system’s thinking) is generating.

Purists would say that I mixed too many things together here, Theory of Constraints, Lean, Six Sigma, the names don’t actually mean anything — the idea is to understand what to take note of during the interviews and measurements.

Take note of the time being spent where something could be automated, where people are doing work that is not bringing value in any way to anything. Let the interviewee speak, but during the note taking and the summary phase mark potential improvements with red and green colors — identifying problems and providing possible solutions.

Humans are the biggest bottleneck in IT, everything that can be automated should be automated and done by computers — they are better at it anyway. Computers are much better at doing repetitive stuff, and once all the hiccups are fixed — usually mistakes are not repeated, unlike when a person does it manually.

When the time is right to actually start changing things, the 1950s can teach us a thing or two about doing the change methodologically. The PDCA cycle, or Deming cycle, gives a hint how changes can be done in a methodological way and the change’s feedback used to actually improve the system instead of doing change for the sake of change. (hint: a waste or time/cost and no value).

heathbroders.com/download/switch-for-organizations.pdf

When doing change, do not forget that what actually matters are the People. SWITCH, a book by the Heath brothers, talks about the human aspects of changing ingrained methods in an existing organization. Change is hard, and in large organisations, it is even harder because many forces are working against it, but it doesn’t mean that it is impossible. Everything is possible and everything can be improved, that is more of a Devops-y way to look at things.

There is another section to this presentation that shows an example of various tools and explains how these can be used to improve stagnated processes and create a faster leaner startup machine. That would be left for another blog post though, as this one is already quite long.

Hope you enjoyed the presentation if you were in the audience, and please leave a comment if you actually read this far and have anything at all to say.

Originally published at www.prodops.io.

--

--