Share:

The debate continues about whether percentages are always better to understand performance, rather than counts.

I’ve written a little about when to use counts versus percentages already. But with the persistence of debate about when to use which, it needs a deeper dive.

In the media and in politics, counts are often preferred over percentages. Perhaps our brains process counts better. Or perhaps counts make size much more apparent. And certainly there will be some that choose one over the other for dramatic impact or influencing power.

Road accident statistics are a case in point. There has been endless debate in the media over recent months about the apparent dramatic increase in motorcycle deaths on Australian roads. Those who use counts of deaths claim a 42% increase in the number of deaths. Those who use percentages of deaths relative to registered motorcycles argue that there has been a 35% decrease in the rate of deaths over the same time period. Australian Motorcycle News put a spotlight on this debate.

Why does it matter? It matters because it’s metrics like these that influence policy and funding. If it’s true that motorcycles are now at higher risk than before, then it warrants more funding to understand why and do something about it. But if it’s true that motorcyclists are safer now than before, why take money away from a more needy cause?

I believe that measurement is to help us get closer to the truth, but it won’t ever be the full truth. When we use counts and percentages, it’s therefore vital that we understand the limitations of each, and the nature of the question we’re trying to answer with them.

Counts are simple to produce and make it easier to comprehend size. When our focus is to reduce the existence of something we don’t want, or increase the existence of something we do want, a count might be all we need. We don’t want more motorcyclists dying on the roads. But counts do have some very specific limitations:

• Counts ignore context, such as the change in population size or opportunity for
  occurrence of the thing being measured. This means an increase in counts might not mean an increase in likelihood.
• Counts are easy to use in limited comparisons, where the two counts we choose to compare to one another can lead to contradictory conclusions. This quarter might be worse than last quarter, but over the year things might be better.

Percentages do account for context and are less sensitive to limited comparisons (but not immune to them). When our focus is to decrease the chance of something we don’t want, or increase the chance of something we do want. We do want to reduce the risk of any motorcyclist dying on the roads. But percentages also have some limitations:

• Percentages look credible, even when based on low amounts of data. If we have 10 customers in our survey and 4 of them are dissatisfied, that’s 40%. With a proper sample size the result could be very different.
• Percentages give less dramatic changes over time (due to the limitation to the 0 to 100% scale), and need more careful analysis to detect real change. If you have 99% system uptime, tracking for real changes needs more sensitive signal detection.

Are percentages always better than counts to measure performance? Often they are, but not always. It’s more important to understand the thing we’re trying to improve, and triangulate measures to get convincing evidence about it.

Percentages don’t ignore context like counts do, but should we always replace counts with percentages to measure performance? [tweet this]