Share:

Quality is a vague concept, which makes it hard to measure directly. Here are 5 steps to more easily measure the quality of anything.

We don’t measure a recipe; we measure each of its ingredients. And the ingredients will vary depending on the recipe and the tastes of the people who will be fed by it. Some ingredients need to be measured because they matter, like eggs in a soufflé, or sugar in cookies, or butter in pastry. But other ingredients don’t need to be measured, like lettuce in a salad, or broth in a risotto, or decorations on a cake. A recipe is a concept and it’s only the ingredients that can be measured meaningfully.

Because quality is a concept – and too vague to measure directly – we have to unpack its ingredients before it we can meaningfully measure it. Inspired by Douglas Hubbard’s clarification chain, from his brilliant book How To Measure Anything (and discussed in this article about measuring design success) is this basic guideline I use to make vague concepts measurable:

1. unpack the multi-focus concept into singular-focus attributes
2. reword the intangible concept into observable language

Quality is both intangible and multi-focus. We can’t meaningfully measure it until we unpack its attributes and make each attribute observable. Here’s how…

STEP 1: What entity do you want to measure the quality of?

Quality relates to so many things. Fundamentally, quality is about how good something is. And even more specifically, it’s about how good something is relative to how good it should be.

So it can be useful to measure the quality of all kinds of entities we work on, like these:

• Data quality e.g. financial data or performance data or HR data
• Product quality e.g. a vacuum cleaner or gourmet chocolate
• Service quality (or customer experience) e.g. of a train trip or legal advice
• Software quality e.g. a meditation app or KPI dashboard app
• Process quality e.g. technical support process or delivery process or manufacturing process

Our first step, then, is to clearly identify the entity we want to measure the quality of. What’s yours?

STEP 2: Who defines the quality of that entity?

I first learned how to measure quality in customer service. My job was to create a system of consistent customer service quality surveys across the railway I worked for. Managers had been defining the attributes of service on behalf of customers. They spent tens of thousands of dollars on surveying customers based on these manager-defined attributes. And they got it wrong.

On-time running of trains was not the most important attribute to customers, as managers had assumed. And the attributes that were most important to customers – safety and security on trains and in stations – the managers were not even aware of. It’s customers that define the most important attributes of quality for the entity of customer service. Not managers.

And for the quality of your entity, you need to let the experts define the quality attributes. And often the experts are the end users, for example:

• Data quality might best be defined by data scientists or others who both analyse data and understand the structure of data
• Software quality might best be defined by the users of the software
• Service quality is best defined by the customers that directly experience that service

Who is the end user of the entity you want to define the quality of?

STEP 3: What’s their overall assessment of quality?

Don’t add up attributes to create an overall. This a composite measure or index and it’s a very unreliable and inaccurate way to measure something. Especially when you can actually measure it directly. And often, we’re too quick to assume we can’t measure something directly, when in fact we can. Here are some examples:

• Overall Perfect Data Rate: using samples of data records, take the percentage of all data items that are either missing, incorrect, incomplete or irrelevant
• Overall User Perception of Software Quality: the average rating provided by users, on a scale of 1 to 10, of the degree to which the software meets their known requirements
• Overall Customer Perception of Quality: the average rating that current customers give, on a scale of 1 to 10, of how satisfied they are with overall service quality

How could you get a direct measure of the overall quality of your entity?

STEP 4: Which specific attributes of quality matter most to them now?

When I was involved in the railway customer surveys, we used focus groups of customers to discuss what quality of service meant to them. And from the qualitative discussion, we listed about a dozen or so quality of service attributes.

They were different for each service: commuters versus freight customers versus tourists. So we can’t rely on finding the “one right answer” someone has already published. But it’s okay to use existing frameworks as a good starting point, or some prompts to start the discussion:

• Data quality – start with these 5 dimensions of data integrity.
• Software quality – the ASQ have this neat checklist of software quality attributes.
• Service quality – a tool called SERVQUAL uses 5 widely accepted service dimensions that most customers care about.

What method could help you get an objective list of quality attributes for your entity?

STEP 5: Measure the most important 3 attributes.

When you have worked out which are the top 3 attributes of quality for your entity, and have made sure you express them measurably, you can go ahead and design your own performance measures of those quality attributes.

I recommend designing your own measures, rather than copy them from published lists of what others measure. That’s because you want the right KPIs that will fit your situation the best. But just for inspiration:

• Software quality – check out Rainforest’s map of Essential QA Metrics for software, including escaped bugs, test coverage, test reliability, time to test, and time to fix.
• Data quality – Christopher Tozzi wrote this list of 7 metrics to measure data quality, which I don’t think is comprehensive but can at least give you a start.
• Service quality – for a freight company’s billing team, how they translated one of the most important quality attributes for their customers into quantitative KPIs.

Just like for any weasely, vague, broad or intangible concept, quality can be measured meaningfully. The trick is to take a deliberate approach to unpacking its defining attributes, expressing each quality attribute measurably, and designing measures as direct evidence of each quality attribute.

Quality is too vague to measure directly. But we can unpack quality into attributes that can be directly measured.
[tweet this]

DISCUSSION:

Use this Make Quality Measurable Template to plan your mini research project to unpack the measurable attributes of quality for the domain you’re interested in.