The last time you ordered a latte, did the barista tell you it would be between three and five dollars? Of course not! So why should website price tags be different? To answer that question, let’s consider the cost of something more familiar–a road trip.

Road Trips

How much would it cost you to road trip from Denver to Phoenix? The most natural approach would be to work backwards: knowing the total miles and your car’s MPG, I could calculate gas costs. For lodging, I could determine which hotels to stay at, for entertainment, settle on a per-diem, and for incidentals, build in a buffer. Add up the costs, and voila–we’d have a price tag. Right? Wrong.

The problem with this approach is there are a thousand things that can go wrong. What happens when you blow a tire and miss your hotel reservation? Stay an extra night at Aunt Mildred’s? Run out of gas? Get in a minor accident? And so on.

Websites

Web and mobile projects are just as fraught with pitfalls, but there’s a further difficulty: often, we don’t even know the destination. We usually know the general direction–for example, you want more conversions or traffic–but the end result, where we’ll end up exactly, is undecided. Projecting the cost of something that vague is like projecting the cost of a road trip with the theme “Go West”: good luck being precise!

I’m exaggerating a little, of course–sometimes stakeholders do know exactly where they’re headed. But even if that’s true, there are still hundreds of decisions that await the project team, not to mention a good many hidden challenges. How complex will the design be? Will it be approved by stakeholders on the first revision or the seventh? How many layers of permissions does your content authoring experience need? How accurate does your document search need to be? How deep should Salesforce integration go? And so on. As a consultant put it to me once, at the beginning of a project, everyone is at the point of maximum ignorance.

The problem, of course, is that the beginning of the project is precisely when you need an estimate. You have to build stakeholder buy-in around a budget and secure funding, and the success or failure of the project may hinge on staying within budget. So how can we get anything resembling a realistic estimate?

The Less-Right Way

The most common way of estimating a large web or mobile project is to break the project into chunks and ask senior team members to put hours on each of the chunks. While straightforward, this approach falls afoul of at least three major problems:

• Known Unknowns & Unknown Unknowns. There are probably big chunks of the project we don’t yet know about, so if our estimate is based only on the chunks we can think of at the outset, we’ll come up short. I see this happen a lot with third-party integrations–we may know a “chunk” of a project is Salesforce integration, for example, but it’s only halfway through the project we realize an entirely new chunk–data cleansing–is required to talk to salesforce correctly. Estimates that don’t in some way account for these “unknown unknowns” (as Donald Rumsfeld famously put it) will always fall short.


• Inaccurate Expert Estimation. When asked to estimate how long it will take to do a chunk of work, experts such as Senior Developers often consider how long it would take them to do something–rather than the person who will actually do the work. Because experts are, well, experts, they work quickly. That means their estimates are a high-water mark of efficiency, rather than a realistic prediction of velocity.


• Planning Fallacy. And even the experts are bad at estimating their time. In their Nobel Prize-winning work, Kahneman & Tversky demonstrated that most people underestimate how long it will take them to do a task. The more chunks an estimate has, the more the effect is compounded, as each chunk provides a new chance to underestimate.

Surely there’s a better way?

Reference Class Forecasts

As a result of their work, Kahneman & Tversky proposed a new way of doing estimates called reference class forecasting. The basic idea is that instead of starting with brand new estimates for each chunk of the project, we start with the actual, total cost of a similarly-sized project. Pure historical data, the thought goes, doesn’t fall prey to optimism or need an ego check. The technique has proven quite effective, and although its application is relatively straightforward, it is not yet commonly used in digital project management.

To see how it works, let’s say you ask me what it will cost to redo your university website. I would start by aggregating data about what it usually costs Aten to redo a university website (the “reference class”). For the purposes of this blog, let’s say that’s 1200 hours. The next step is for me, as a subject matter expert, to adjust this number up or down based on my understanding of the project–so for example, the types of content you’ll need, the number of content pieces, and so on. For this example, I’ll adjust the total downward to 1000 hours. The final step is to determine how certain I am about my ability to predict these costs (say, 80% certain) and that will yield a range, which is the final answer: 800 - 1200 hours. That's it!

So to answer our central question, a website will cost about as much as it last cost to make a similar website.

Or at least, in order for a vendor to make money–that’s the minimum they should charge. In practical terms, this means that as a project sponsor, you should keep a few things in mind when evaluating proposals:

• Don’t be afraid to ask questions. What method did an agency use to arrive at the estimate they did? Did they use historical data? Does their culture warrant high confidence in that data? Why is your project more or less than similar projects they’ve completed? How do they account for known & unknown unknowns?


• Look for ranges. The best estimates come in ranges–a pair of low and high numbers–because the future is uncertain. Ranges are sometimes seen as the result of laziness or imprecision, but in fact, when correctly used, ranges are a marker of realism and precision–which is exactly what you want in an estimate.


• If you have a fixed budget, be prepared for a variable scope. Because we work with cause-driven organizations, many of our clients come to us with a predetermined, fixed budget. We welcome these kinds of engagements and are clear at the outset that we'll work with you to get a great product out the door, but we can't promise you'll get everything you ever wanted. Look for vendors who practice this kind of straight talk–it's essential for a successful engagement.


• Select a partner, not a contractor. No matter how well-defined you think the scope of a project is at its outset, it will change. What this means is you need an experienced, strategically-minded partner who can help you adapt, rather than a contractor who wants a change order at every turn. Aten, it happens, is just such a partner!

A Final Note

Estimates are really hard to get right. By definition, they’re created out of incomplete information, which leaves plenty of room for sunny, but ill-fated optimism. Beware the agency that is overconfident or promises too much for too little; instead, look for honest assessments of risk and uncertainty. In the long run, selecting a partner that takes a disciplined, realistic approach to project management will pay huge dividends.