Perspectives · Perspectives
Sustainable Estimating
Peter Milsom January 15, 2018
The only certainty is that nothing is certain. Pliny the Elder, Roman scholar, 23-79 CE
Our culture encodes a strong bias either to neglect or ignore variation. We tend to focus instead on measures of central tendency, and as a result we make some terrible mistakes, often with considerable practical import. Stephen Jay Gould, naturalist, 1941-2002
This post establishes the importance of sustainable estimating techniques. These techniques can be incorporated into risk management and business case estimating as well as scheduling within sustainable change delivery. This blog post is part of a series that provides the foundation for understanding sustainable change delivery.Context
The Risk of Usurping the Sponsor's Authority
I frequently see situations where project teams provide the estimated duration, resources and cost to complete a project as just a series of individual numbers (i.e. six months, five resources and $500,000). Each of these numbers represent a range of numbers each with its own probability of being met. In many industries and organizations, sponsors could not even be advised what the impact of a change in the estimate would do to that probability... if indeed they knew what the probability of success actually was. In short, sponsors are often not provided with the full range of choices in project estimates. Due to this, project teams are usurping the decision maker's authority to decide how much risk to take on and what they'll accept as the probability of success or failure. Giving decision makers a complete picture of estimate uncertainties is vital to effective planning. Telling your sponsor that the project will be done in six months, with five resources and $500,000 leaves out important information and usurps the manager's right to manage risk (i.e. that there is only a 40% chance of success). This post will explain how to deal with this.Understanding the Dangers of "The Flaw of Averages"
The flaw of averages states: Plans based on average assumptions are wrong on average (Savage, p.3, 2012). A classic example of the Flaw of Averages involves the Statistician who drowned crossing a river that was on average 3 ft. deep, as shown in exhibit 1. This is the classic problem with averages, expected value or central tendency. Here is another simple example."Imagine that you and your spouse have an invitation to a ritzy reception with a bunch of VIPs. You must leave home by 6 p.m. or risk being late. Although you work in different parts of town, each of your average commute times is 30 minutes. So if you both depart work at 5:30, then you should have at least a 50/50 chance of leaving home together for the reception by 6 o’clock. Suppose there really is a 50/50 chance that each of you will make it home by 6:00. Then the trip is like a coin toss in which heads is equivalent to arriving by 6:00 and tails to arriving after 6:00. Four things can happen:Before getting into the specifics, here are some useful perspectives to demonstrate how we should evaluate estimates for tasks, impact, and probability as a range due to the uncertainty.The only way you can leave by 6:00 is if you flip two heads, for which there is only one chance in four" (Savage, p.2, 2012).
- Heads/tails: You are home by 6:00 but your spouse isn’t.
- Tails/heads: Your spouse is home by 6:00 but you aren’t.
- Tails/tails: Neither of you is home by 6:00.
- Heads/heads: Both of you are home by 6:00.
- Assumptions (Point Value):
- Best case or ideal value (no uncertainty) provides a fixed number (i.e. exactly 1 day for a meeting).
- However, in Reality:
- Most activities in Project Schedules are represented as exact values even though exact values are almost never known !
- Or worse, they are provided a single number which is perceived as an average !
- No Assumptions (Ranges):
- Most things we DO know are better represented by ranges and probabilities – we don’t have to assume anything we don’t really know.
- This is represented as a “threshold confidence’.
Recommendations
Approach
The first step is in identifying the work or product breakdown structure, often organized by work packages. For each activity a subject matter expert should provide an appropriately detailed explanation, and then start working on the estimated duration. If appropriate ranges should be provided by calibrated experts that also provide distributions, which can then be run through simulations software using Monte Carlo or another simulations tool. The following is a modification of the Humphrey approach to earned value project management from project initiation until the integrated baseline plan. The oval entries represent our recommended modifications to provide sustainable estimations.Estimator Calibration Training
In order to get the appropriate activity duration estimate though, people have to be trained to provide calibrated estimates. For all estimates, teach subject matter experts and risk owners to provide an upper and lower bound that they are 90% certain contains the correct answer. This normally only takes a couple of hours. The challenge is that decades of studies show that most resources are statistically “overconfident” when assessing their own uncertainty. Curiously, studies have shown that bookies were great at assessing odds subjectively, while doctors were terrible as were young professionals just out of school. Fortunately studies also have shown that measuring your own uncertainty about a quantity is a general skill that can be taught with a measurable improvement. Training can “calibrate” people so that of all the times they say they are 90% confident, they will be right 90% of the time. Douglas Hubbard’s book The Failure of Risk Management provides training on calibration, and he also provides a course on this topic. The next two graphs came from Douglas Hubbard's research on the topic of calibration. He discovered that most people are significantly overconfident about their estimates, especially educated professionals: However, as indicated in both the previous and following graph, people can be trained to provide better calibrated estimates to the 90% level: Unfortunately, almost nobody uses these methods. Calibrated Estimating – Calibration Aids Here are some questions to assess your estimate:- “The Equivalent Bet”: for 90% Confidence Interval questions, which would you rather have?
- A: Win $1,000 if your interval contains the correct answer
- B: A 90% chance to win $1,000
- Think of a couple of pros and cons for your range. How could it be wrong? Why do you think it is right?
- Being "right". Are you focusing on being “right” instead of realistically representing your uncertainty?
- Are you hanging on to traditional expectations of “+/- 10%” or similar narrow ranges? Are you resisting wider ranges because you think they are “too wide”?
- Equivalent bet. Are you actually trying an equivalent bet on each activity?
- Try ways to avoid “anchoring”. Don’t think of one number then add and subtract an error. Instead, treat each bound as a separate binary question (e.g. are you 95% certain the value is less than the upper bound?)
- Control the “Story Teller”: There is often a strong temptation to explain in detail complicating factors, exceptions, historical background, etc. Resist the temptation to explain in detail why you have uncertainty. It’s a given that you have it. Provide the range.
- Remind them to not assume wide ranges are useless: If it represents your uncertainty fairly, that’s the range we want. Whether that range needs to be narrowed is another step.
- Resist “Infinite Decomposition”: You can always compute a value based on other more detailed values but at some point you have to just provide a range.
- Avoid Group-think: There is a tendency for individuals in a group to adopt the confidence of the persons who talk the most. Prompt the quiet to speak. With some groups, individual estimates obtained by email is best.
- Remind them that they are calibrated: Their performance skill at assessing odds has be proven quantitatively.
- Redirect!: When the “explanations” seem to go on too long and they avoid the estimate, prompt them with a question – “Is it more than X?, Less than Y?”
Task Distributions
Once the task has been identified, the range and probability figured out, the estimator needs to be able to provide a distribution for the activity, or an arrangement of values of a duration range showing their theoretical frequency of occurrence. Take the low, median and high duration dates, and assign the distribution occurrence that makes the most sense. For example:Employ a Monte Carlo Simulation Tool
A Monte Carlo simulation is a computerized mathematical technique that allows project teams to account for risk in quantitative analysis and informs decision makers. Monte Carlo simulation furnishes the decision-maker with a range of possible outcomes, and the probabilities they will occur, for any choice of action. It shows the extreme possibilities—the outcomes for the most conservative decision makers, or the outcomes for the riskiest, along with all possible consequences for middle-of-the-road decisions. These can help guide project teams and sponsors on what options will provide the greatest chance of project success based on the scheduling model and actions input into the model. Monte Carlo simulation performs risk analysis by building models of possible results by substituting a range of values—a probability distribution—for any factor that has inherent uncertainty. It then calculates results over and over, each time using a different set of random values from the probability functions. Depending upon the number of uncertainties and the ranges specified for them, a Monte Carlo simulation could involve thousands or tens of thousands of recalculations before it is complete. Monte Carlo simulation produces distributions of possible outcome values. By using probability distributions, variables can have different probabilities of different outcomes occurring. Probability distributions are a much more realistic way of describing uncertainty in variables of a risk analysis. Following are a very simplified series of images for a Monte Carlo tool called Risk +. Personally I prefer Barbecana's Full Monte or Palisades @Risk.Inputs
The entry screen is normally fairly straightforward. One identifies the activity name, minimum effort, most likely effort, maximum effort, and the nature of the distribution curve:
Simplified examples of the distribution curve are listed below:
The option for the number of iterations helps to provide a greater sample and reduce uncertainty. It also helps to chose which analysis option is desired, which will be described in the following images.Outputs
If the schedule and costs acceptable probability do not align with the organizations project constraints, such as completion date or overall cost, are there activities with ranges that need to be removed or tightened up or analyzed further to reduce uncertainty and fit within the known constraints?
The following two images provide an example of potential / possible completion dates and cost based on probability. The information for the sponsor is that the current plan has a 50% chance of success of meeting the February 20 20## date AND of spending 598K or less AND delivering all of the project outputs… i.e. the probability of meeting or improving on that estimate.
The following is an example of potential / possible completion costs based on probability:
The following is an example of a sensitivity analysis, which highlights those activity ranges that may require more research and refinement:
Sensitivity analysis reinforces the value of information, showing what activities require more research to improve understanding and estimates.
