Share:

Does your team spend more time debating the integrity of their KPIs than they do using them to improve performance?

It always frustrates me how much time decision-makers spend arguing about the accuracy or precision or integrity of a measure, compared to the time they spend using that measure to inform decisions.

The terms accuracy and precision and integrity are often used interchangeably when we talk about KPIs. Yes, it’s important that our measures have enough of these things so that they don’t lead our decision-making astray. But how much KPI accuracy and precision and integrity do we need?

For a practical answer to this question, we need to first deweasel these terms: accuracy and precision and integrity.

Accuracy means your measure’s mean is close to the truth.

Imagine you’re shooting arrows at a classic bullseye target.

Accuracy is akin to how close, on average, your arrows will land relative to the centre of the bullseye. A measure is more accurate the more directly it is measuring the result you are trying to measure.

Average Customer Satisfaction Rating is a more accurate measure of customer satisfaction than Number of Customer Complaints is. That’s because customer complaints usually only come from the “squeaky wheels”, and they drown out the valid and important and balancing views of all other customers.

Squeaky wheels, volunteer surveys and easiest-ones-to-measure are examples of data sources that render your measures inaccurate. Your measures end up biased, which means they centre around a point that’s different to the bullseye.

Precision means your measure’s variation isn’t unnecessarily large.

Precision is akin to how close to each other your arrows are when they land on the target. A measure is more precise when it is based on enough data to get a reliable picture, as you measure it again and again over time.

Would you rely on one customer’s satisfaction rating to draw conclusions about all customers’ satisfaction? What about five customers? How many customers would you need to get a precise enough estimate of Average Customer Satisfaction Rating? There isn’t one right answer, because it does depend on your customer population and how variable their ratings are likely to be. But we can thank the survey statisticians for developing a method to calculate the best sample size in different situations.

Truth be told, KPIs or performance measures of any kind are never precise. Rather, they have a degree of precision that depends on the sample size for collecting the data, how that sample is selected, and how meticulously the data is collected and recorded. In deliberately designed data collection regimes, precision can be measured (e.g. using confidence intervals). If your measure’s data isn’t reliable, it means its calculations can vary wildly around the bullseye.

Minitab have an article that dives more deeply into accuracy and precision, with an excellent bullseye graphic to show their relationship to one another, and to the truth.

Integrity means your measure is accurate enough and precise enough for its purpose.

Integrity, in the context of KPIs, means your measures have enough accuracy and precision for your needs. And, from my background as research statistician, integrity can be unpacked into five actionable components that help us make practical decisions about getting enough accuracy and precision:

Dive more deeply into the five “Rs” of data integrity here. And make sure you follow proven data collection design steps, to get the best data you can afford for each KPI.

How much accuracy and precision do your KPIs need?

A measure can be precise and accurate. But it can also be accurate but not precise, or precise but not accurate. Accuracy is mostly influenced by how well we design our
performance measure and it’s underlying data. Precision, however, is influenced by how much and how meticulously we collect a measure’s data.

If you ask me, I treat accuracy as more important than precision. Partly, because precision is more costly than accuracy. And mostly, because it doesn’t make sense to measure something inaccurately, with high precision. But we can still make sound decisions when we measure something accurately, with moderate precision.

Stop wasting time debating the integrity of your KPIs. Make them just accurate enough, and use them to improve performance.
[tweet this]

DISCUSSION:

Do you have the problem of people debating KPI integrity, and never getting to the point where the KPI is used to improve performance?