Share:

Sampling the data for your KPI not only takes less time, energy and cost, but often it’s more accurate.

Too many great KPIs or performance measures never see the light of day because of beliefs that the data will be too hard, too costly or too time-consuming to get. Not so!

I started my career as a survey statistician and one of the powerful tools we used was sampling. I used samples to measure all kinds of things, such as:

• the profitability of broadacre agriculture farms across Australia
• the economic impact of motorsports on the Gold Coast
• the accuracy of inventory records in a railway

These measures would have been impossible to implement, with any level of reliability if at all, without sampling.

Samples, believe it or not, can produce more reliable data than populations.

It’s easy to appreciate that samples are a far less expensive data collection effort than collecting data from the entire population. But, ironically, measuring with samples is often much more reliable than collecting data from entire populations.

For example, trying to visit every broadacre farmer in Australia, to survey their profitability would have cost a fortune in travel. And there wouldn’t be the luxury to spend enough quality time with each farmer. I was often invited to join them for dinner in my visits, and it helped build the rapport to gather the data in a careful and respectful way.

And with the case of measuring inventory accuracy in the railway, a sample helped them discover that their inventory records were much more accurate than their previous data collection method suggested.

When we use samples, we can afford to spend more time on designing a data collection process that preserves the integrity of the data.

The problem with sampling is the misconceptions about how to do it.

Sampling is used by lots of people who aren’t qualified statisticians, and that can be just fine. But too often their approaches include some common mistakes in using samples, such as:

• sample sizes need to be at least 10% of your population size (not true – sample sizes depends more on the variability in our measure values and on the level of precision we want)
• sample sizes need to be maximised to give reliable enough results (not true – we can opt for smaller sample sizes to save costs if we’re happy to live with lower precision in our measure values)
• volunteers or hand-picked samples are reliable (not true – even though they are convenient, for quantitative measures a randomised sample will have less bias)
• important measures shouldn’t be based on samples (not true – even though we will have sampling error in the measure values, tracking changes over time can still be very accurate)
• a qualified survey statistician or market researcher is always needed (not true – we can learn the basics of designing simple samples)

Don’t fall for these misconceptions. Be informed and you’ll save a tonne of time and energy (and decision-making error) that would otherwise be the result of a poorly designed sample.

How to know when you can try sampling for yourself.

It’s true that for complex measurement problems, we need the expertise of a qualified survey statistician (it’s a profession: I studied at university for 4 years to become one, then worked in the field for 3 years and still had lots to learn).

For example, the railway’s inventory accuracy measure that I created a sample design for was very complex. We first sampled inventory sites, using stratification to make sure we included a representative range of the highly variable sites. Then at each site we sampled inventory line items, again using stratification to make sure we got the full spectrum of highly variable inventory types. This makes sample size calculation challenging, and you need rather complex mathematical formulae to calculate the inventory accuracy measure values. Not for novices!

However, for simple sampling, you might like to try it yourself. If you’re into DIY, here are a few guides to do sampling the right way:

• how to design reliable sample sizes
• how to select a simple random sample
• how to design good survey questions or constructs

It should feel like a revelation that we can use proper sampling to save heaps of time and cost, and bring to life those measures that really matter. So, when you don’t think your ideal measures are feasible, think again!

Using sampling to get KPI data is like a blood test, it can be reliable and cost-effective, but only when it’s done right. [tweet this]