Okay, so I’m just going to go ahead and say it:
It is impossible to accurately estimate a software project of any significance.
Now, a non-trivial number of you are going to read that sentence and think I’m nuts. And maybe I am. But someone has to just say what we all know to be true but don’t want to admit.
Look, there have been countless books written, innumerable conferences held, untold consulting hours purchased, and endless blog posts written on how to be better at estimating software projects. I get it. We all work earnestly to give our best effort in an attempt to placate hungry bosses who want to know when a new feature will be ready. We all set deadlines based on a conference date and not when the software will actually be ready.
But the bottom line is that we simply can’t do it. Well, we can do it—indeed, we do it all the time—but we can’t do it well. In other words, we are always wrong.
I mean, we keep spending the money and going to seminars and reading the blogs and books. We bring in highly paid consultants who act like they know what they are talking about. But it just never gets better. We suck at it, and we keep thinking we can improve and that the next fad really will be the silver bullet. But we won’t admit to ourselves what we know to be true: We can’t estimate software projects with very much confidence at all.
We’ve all done projects that we thought would take a given chunk of time and then they end up taking twice or three times as long. You may have been involved in a project that finished in half the time expected. But it is a rare and strange project that finishes within a tight window around the actual, original estimate. And then, when we apply Hofstadter’s law (“It always takes longer than you expect, even when you take into account Hofstadter’s Law”) and double the estimate, that proves to be wrong, too.
There are reasons this is the case. The most prominent one, and the one I think most folks have the hardest time with, is that each and every software project is unique with its own long list of “unknown unknowns.” You can plan, strategize, chunk, fold, spindle, and mutilate a project for countless person-hours, and you still won’t know the difficulties that lay ahead in actually writing the code. Some things that you think will be challenging will turn out to be easy. But most often, you will dramatically underestimate the challenge that some particular aspect of the project will pose.
Of course, this happens because the average software manager will always believe that the path taken will be a flat, straight highway with lovely weather all the way to the destination. And that is simply never the case. The requirements change, never making the project smaller. People take unexpected PTO. Other projects or priorities interfere. The sales department needs this one little tweak to close a major deal. Nothing is fair winds and following seas from start to finish. Ever.
I have a friend with a fancy degree in computer engineering who gets so mad at me when I say this, but the real problem is that software development is not an engineering discipline. Instead, it is a process mired in changing human desires, interacting personalities and dynamics, shifting customer understanding, and unscientific solutions. Software development is a creative process, not a scientific one, and creative endeavors cannot be distilled down to knowable steps and a repeatable system.
Hey, I get that this is hard to hear. Businesses—and by that I mean customers—don’t want to hear “Well, we really aren’t sure when we’ll have that for you.” They’ll seek out vendors who will tell them what they want to hear, even if it is complete baloney. Companies have to make money, and to do that they need to produce value sooner rather than later. That demo of the new feature actually does have to happen at that conference on that particular date.
And we need to figure out how to accept and live with this. I say this because I think that as an industry, we have been on a decades-long quest for the holy grail of software estimation, and we always will be. But we’ll never figure it out. We just won’t. Until we come to terms with that, we’ll struggle and flail and continue to tell ourselves something that we know simply isn’t true.
I don’t have a solution, and I doubt there even is a solution. Accepting that is the first step in coming to terms with a problem that simply will never go away.