Blog

Don’t bury the waterfall

17 Dec, 2021
Xebia Background Header Wave

TLDR: if you do sprints you might not be actually doing Agile. At the same time, separating your process into development and delivery phases doesn’t necessarily mean you’re doing it wrong. However, hiding the actual methodology you are practicing under labels may do more harm than good. And if your organization’s development process performs linearly, don’t bury it under the agile elements. 

Introduction 

In modern software-related discourse, “waterfall,” “monolith” and “yearly releases” are synonyms for failure. It seems that, unless you’re doing some variant of agile development (even in name only 😉), you’re doing it wrong.  

So why is that? And is “waterfall” a straight road to disaster as it is so often portrayed?  

To find out, let’s first look at the early years of software development. 

Down (a very simplified) memory lane 

The first software programs were written, not by Computer Science graduates, but by electrical engineers. These engineers didn’t use IDEs or copy from Stack Overflow – the earliest programs were carefully designed by hand, typically considering the circuitry and configuration of a particular computer system. After writing and validating the algorithm and operations, it was time to run the program. Execution took a lot of time as well, (though still faster than was humanly possible) and the computing time was shared among colleagues and peers. Thus, if your program made mistakes or crashed, you had to go back to the drawing board and retry later when it was your turn in the queue again.  

Fun fact, but the first patching of a software program was a literal patch on a punch card hole.

A program tape for the 1944 Harvard Mark I, one of the first digital computers. Note physical patches used to correct punched holes by covering them. (CC BY-SA 3.0)

And the first bug was a literal insect found in between the circuits.

A page from the Harvard Mark II electromechanical computer’s log, featuring a dead moth that was removed from the device. (Public Domain)

Back then all these elements (planning, design, implementation, testing, and execution) took a lot of time. Since the cost of mistakes was quite high, careful execution was key.

Naturally, when writing programs became more involved and required an oversight project effort, delivered by groups of people, the chosen development methodology reflected that. It copied the careful and thorough approach to development that is normally found in real-world engineering jobs, where the cost of mistakes is high, i.e., used electricity, wasted time, and lost physical materials.

Classic stages of a “Waterfall” process

A “Waterfall[1]” methodology follows a linear flow, normally comprising:

  • Collecting the requirements and making sure they are complete
  • Designing the software conforming to the requirements
  • Code implementation
  • Verification of the results (usually with acceptance testing by the user)
  • After that, the software enters the maintenance phase (where, on average, 80% of the money spent on the software is consumed).

[1] An important note – Dr. Winston Royce, who created the classic Waterfall model, did include opportunities for feedback (Royce, W. W. Managing the Development of Large Software Systems. In IEEE WESCON, pages 328–338, Los Angeles, 1970. IEEE.).

New Dawn Fades

This approach showed its flaws when software development started to become more and more complex – it started to engage with this new thing called “the Internet,” being built on top of different technologies, and sometimes even requiring integration with third-party software packages.

And so, the problems of long planning and implementation met with the speed of technological advancements, which often resulted in:

  1. The released product’s use case having changed by the time it was released.
  2. The inability to respond to the competition during the development cycle and adapt quickly.
  3. Communication overhead between multiple involved parties requiring many handovers, resulting in loss of context and the introduction of assumption, causing many bugs and flaws to appear late in the process.

At the same time the technology, computers and human resources became widely available and execution cycles got cheaper. Next to that, software development started to slowly embrace concepts such as automated testing, automated provisioning, and version control, culminating in the realization:

Making mistakes during software development no longer has to cost you a lot of money, provided you catch, validate and fix them quickly

Welcome to Agile

Agile practices officially came into mainstream software culture at the beginning of the century with the publication of the “Agile Manifesto” in 2001. While the underlying ideas were not new, it was the first successful movement that provided clear focus in earlier practices like Kanban, SCRUM, Extreme Programming, etc. It put forward prioritizing collaboration, delivering functionality and flexibility in planning over rigid planning, extensive documentation and reliance on tooling and process.

Most agile methodologies are staying true to these principles. The work is delivered in iterations, allowing more frequent user feedback and course correction throughout the development cycle. Certain methodologies, like SCRUM, focus on releasing a functioning product increment with every iteration, however small the development team can make it – the important part is that it is released.

Essential elements of Scrum methodology

Since then, Agile practices rightfully became the “default” choice for software development projects due to their apparent benefits. Stakeholders are able to see the gradual delivery of the working end product with every iteration, while developers are able to course correct, improving their internal and external collaboration and minimizing the post-release work.

So far so good, right?

As you probably know yourself, in the real world many projects are still delayed / have problems with feature launches, users don’t always get what they asked for, and bugs still end up in production. So where is the problem?

Reality kicks in

There is, of course, no single issue that causes mishaps in software development, yet when it comes to methodologies a common one can be summarized as follows:

A methodology is way too often applied as-is, without taking a holistic view of the needs of a company, the team, the product being built, or the requirements of the product. The methodology (especially when it’s hyped and shiny) becomes a “silver bullet” that “will make us the next FAANG[2]” and is applied broadly to every aspect of the company, and often never re-considered again.

Seven years ago, every second post on LinkedIn tried to include “blockchain” when describing their product. Afterwards it was “microservices,” then it was “machine learning,” and in 2019 it was “AI”. Of course, some of it can be explained by a marketing department making sure to keep the company in the discourse by keeping up with the latest trends, but if “AI” brings no benefit to your business functionality, leave it out of your next product-goals meetings.

The same applies to a methodology – when it is just pushed onto a team without prior research (or because it has a “cool name,” like the Spotify model, which was never actually used at Spotify), it becomes a burden on the team. Ask yourself:  “have I ever encountered any of the following in my day-to-day life”:

[2] Facebook, Apple, Amazon, Netflix, Google

  1. “We’re doing “Agile,” but the release was already scheduled to be after 3 sprints, and we haven’t even started UAT”
  2. “The requirements for our service are crystal clear, yet we continue to demo our product to stakeholders, most of whom are not present, and the others will never interact with the service anyway!”
  3. “We would like to start working on it, but IT needs to align with people from Operations, and Patrick is on leave until next month, so we’ll do a maintenance-sprint this time”
  4. “When we are done with the implementation sprint, the testing team will start the verification sprint and we’ll get the results after that for improvement!”

It happens, when a company operates in one methodology, but “pretends” to be something else. The most basic example is when you turn every step of the waterfall process into a collection of sprints for the sake of it being “agile.”

In the end, you get the benefits of neither and drawbacks of both.

“How do you do, fellow agilers?”

So, when no methodology is a silver bullet, how do you choose and apply one for your team? Let’s lay some facts on the table.

Agile, Waterfall and Your team

Looking broadly at Agile and Waterfall, the differences can be summarized as follows:

  • Agile shines when you need to discover things as you go
    • … while Waterfall works well if the requirements are crystal clear from the start
  • Agile allows you to collaborate efficiently with stakeholders and gather feedback during demos throughout the delivery
    • … but if no changes to requirements are expected, Waterfall saves you time
  • Agile puts the responsibility on the developers, promoting self-improvement and expecting multidisciplinary teams
    • … while the sequential nature of Waterfall works great with task-oriented groups with established workflows

To identify the most effective approach, ask yourself the following questions before starting a new project:

  • Are my requirements clear?
  • Is my technical stack set?
  • Do I have a set deadline?
  • Is the project’s timeline short?
  • Do I have available resources in the teams with required expertise?

If you answered yes to all of them, “Waterfall” can actually be good choice when executing your project continuously, with a known context, destination and set of tools and resources.

But if you answered these questions with something like:

  • The requirements are only high level, and we will have to try the software first with end-users before agreeing on the end state
  • The technical stack might change if we find out the software doesn’t work as expected
  • No deadline is set until the software takes shape
  • Our team is composed of multidisciplinary developers where everyone is equal in skills

You might lean more towards “Agile”: implement, release, and show the product to your customer, regroup, rethink, and try again.

In the end, each individual situation will pose unique challenges and opportunities, where dogmatic thinking about approaches to software development doesn’t help. And perhaps it is about time that modern companies step away from sticking to one rulebook all the time and take a more “gearbox” approach to delivering software products, shifting in the appropriate gear depending on how fast (or careful) they want to go.

Just make sure you don’t go in reverse 😉

 

Dave van Stein
Process hacker, compliance archeologist and anthropologist, ivory tower basher, DepSevOcs pragmatist, mapping enthousiast, complexity reducer, intention sketcher. LEGO® SERIOUS PLAY® Facilitator.
Questions?

Get in touch with us to learn more about the subject and related solutions

Explore related posts