Share:

We slice, we dice, we tabulate, we chart, we dashboard and then we draw conclusions. But how often have we drawn diabolically incorrect conclusions because we have violated the two basic principles of understanding our measures?

In Donald Wheeler’s book Understanding Variation, he introduces two very simple and sensible principles for understanding data:

1. No data have meaning apart from their context.
2. While every data set contains noise, some data sets contain signals. Therefore, before you can detect a signal within any given data set, you must first filter out the noise.

What do these principles mean, particularly in the context of reporting performance measures or KPIs?

PRINCIPLE 1: No data have meaning apart from their context.

The first principle means that you must understand where your data come from, and you must have enough data, before you can understand what a measure is telling you.

Here are some practical tips to ensure you report your measures, meaningfully, with enough context:

• Provide a link to and a summary of the meaning of the measure: it’s calculation, data source and plain-english description of what the measure means.
• Report more than just one or two values of your measure. In fact, do your best to report your measures in time series, including up to 3 years’ worth of historical values.
• Put your measures into graphs so it’s faster and easier to see patterns over time. When we’re trying to improve performance, it’s the patterns over time that tell the most insightful stories of performance.

PRINCIPLE 2: Before you can detect a signal, you must first filter out the noise.

The second principle means that every time you compute a difference between two values, you cannot assume that difference means there was a change. These computations include this month to last month, this month to the same month last year, this month compared to target, and so on.

We all know how to quantify performance using averages and totals and percentages. But in order to filter out the noise in our measures, we also need to quantify the variation.

Here are some practical tips to ensure you report real signals in your measures (and not be misled by the noise):

• For performance measures you monitor regularly over time, use XmR charts.
• When you are reporting survey results that were based on a sample, report the confidence intervals for the survey estimates.

I’m really looking forward to data visualisation expert Stephen Few’s next book on this very topic, called Signal.

TAKE ACTION:

Take a look at your performance reports and see if you can find any measure or KPI presentation that is violating the two principles for understanding data. What risks does this expose your decision-making to?