Share:

With the help of Doug Hubbard, author of “How to Measure Anything”, we’ll explore why you can abandon the pursuit of perfectly accurate data and measures.

STACEY: Doug, for years now you’ve been coaxing people in the IT field to change their views from “IT is too intangible to measure” to “everything is measurable.” And I can attest that it’s not just in IT that needs to happen. Why has this been a hard transformation to them to make?

DOUG: IT often sees measurement as a choice between perfect 100% certain precision and nothing. Since they see perfect certainty as unachievable, they opt for no measurement at all. They’ve overlooked the usefulness of a third option: the “good enough” measurement.

STACEY: What have you found are the reasons this third option – and in my view often the only option – has been overlooked?

DOUG: Well, measurement doesn’t mean what they think it means. When I ask CIO’s and other IT managers at my seminars what measurement means, I often get an answer like “Assigning a specific value”. This is wrong in at least two ways.

First, in science values aren’t simply “assigned”, they are based on observation. Second, a measurement is seen as a reduction in uncertainty, almost never the elimination of uncertainty. In effect, science uses the term measurement to mean “observations that reduce uncertainty about a quantity”.

IT, on the other hand, thinks of measurement more like accountants think of it: absolutely precise, but often arbitrary and not always based on an observation.

STACEY: So that’s one thing you can do – reframe measurement as a process that helps to reduce uncertainty, not eliminate it. Are there any more hurdles to jump over before they get comfortable to start measuring imperfectly?

DOUG: Yes, and that’s accepting the idea that the presence of noise does not mean a lack of signal. Many managers, IT and otherwise, start anticipating potential errors in any measurement and assume that any existence of any kind of error undermines the value of the measurement.

The fact is that they have to make assumptions about how common these errors are or the effect they would have on the outcomes without having any idea of the relative frequencies of these problems. To put it another way, they have more error in their “error identification” method than the measurement is likely to have.

STACEY: Ha! I like that! It would be a really interesting experiment to find out how big the error in the measurement could get before it drowned out signals in the measure. Wouldn’t that help convince the sceptics?

DOUG: In my consulting practice I compute something the decision sciences call “Expected Value of Perfect Information” (EVPI) and “Expected Value of Sample (Partial) Information” (ESI). Of course, the cost of perfect information would almost always be greater than the value of perfect information. In fact, it is almost always the case that the “biggest bang for the buck” is the initial, small amount of uncertainty reduction in a measurement.

STACEY: Doug, what’s the take-home point you’d like to leave readers with, regarding the imperfect nature of measurement?

DOUG: These three concepts should help IT – and anyone hung up on precision – start to make usefully imperfect measurements. Of course, they require thinking about measurement more like a statistician, scientist, or actuary would and less like an accountant normally would. You should always choose imperfect but relevant measurements over arbitrarily precise and irrelevant measurements.

STACEY: Measures just need to have enough precision to give you the signals you need to make meaningful improvements to the business.

Get the full interview here: www.measureupblog.com/doug-hubbard-interview

TAKE ACTION: How many good measures have you ignored or abandoned because the data wasn’t perfect? Or even what you felt might be good enough? Even when the data is a bit misleading, the overall trend through time can still be very informative and valid. Might be worth asking yourself: is knowing something about how this measure is trending better than knowing nothing at all?