PRACTICAL
PERFORMANCE
MEASUREMENT

Using the PuMP Blueprint for Fast, Easy and Engaging KPIs

STACEY BARR

Copyright © Stacey Barr 2014

PuMP® is a registered trademark of Stacey Barr

Published in Australia by The PuMP Press
PO Box 422, Samford Qld 4520
http://www.thepumppress.com.au
info@thepumppress.com.au

First published in 2014

COPY EDITOR: Nan Wishner
COMPOSITION: Book Matters
COVER ART: Emir Oručević
PHOTOGRAPHY: Laura Maxwell and Melissa Kneipp
PRINTER & BINDER: McPherson’s Printing Group

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval system without written permission from the authors or publisher, except for the brief inclusion of quotations in a review. Requests for permissions should be directed to info@thepumppress.com.au.

The PuMP Press books are available at special discounts for bulk purchases for sales promotions or corporate use. Special editions, including personalised covers, excerpts, or books with corporate logos can be created in large quantities for special needs. For more information, email info@thepumppress.com.au or call +61 1800 883 458.

ISBN: 978-0-9923837-0-1
eISBN: 978-0-9923837-1-8

National Library of Australia Cataloguing-in-Publication Entry:
Barr, Stacey, author.
Practical performance measurement: using the PuMP Blueprint for fast, easy and engaging KPIs / Stacey Barr; Nan Wishner, editor.
ISBN: 9780992383701 (hardback)
eISBN: 9780992383718
Includes index.
Organizational effectiveness.
Wishner, Nan, editor.
658.4

Printed in Australia

FIRST EDITION

Logo for paper and ink from responsible sources

To Mum and Dad, for the rock-steady foundation you gave me.

To Scott, for the most faithful and unconditional friendship I’ve ever known.

To Kevin, for inspiring me to specialise in performance measurement.

INTRODUCTION

That you have this book at all rests upon a presumption.

The presumption is that you believe, or the person who gave you this book believes, that measurement is essential to making an organisation—or company, or business of any kind or size—perform.

Many organisations have performance measures because their business plans have a key performance indicator (KPI) column, or because stakeholders insist, or policy demands it. But performance measurement is so much more than what most of us understand it to be, which is one of the reasons why many of us still struggle to make it work and to realise its true potential.

The true purpose of performance measurement—indeed its real power—is that it helps us achieve more of our goals sooner, and with less effort and money than would be possible without it. It does this by snapping our attention onto the things that matter most to our organisation’s success, telling us in no uncertain terms how well we are (or aren’t) achieving those things, and motivating us to find better ways to achieve them.

Most likely, you have this book because performance measurement isn’t living up to its true purpose in your organisation. You don’t get the information you need from it, your KPIs are not much more than boxes that you check each quarter or each year, or you wish you could get your hands on more meaningful data to help you take effective action to achieve your goals. If you are facing these frustrations, chances are that one or more of the following is true: your measures are meaningless, your organisation can’t figure out exactly how to measure what really matters, you don’t trust your measures’ data, people refuse to own the measures, or performance isn’t improving.

In short, you are probably reading this book because your performance measurement process is broken.

When a performance measurement process works well, measures keep us focused on what we know matters most, people feel enlightened by the measures they use and motivated to improve performance, we know what current performance actually is and why, and we know where performance should or could be and what needs to change to get there. Effective performance measures are key to doing business successfully, and should be a ubiquitous element of our work. They should stop us from stumbling around in the dark to figure out where we are and where we’re trying to go, as well as from bumping unexpectedly into all manner of obstacles along the way.

This book is not going to try and convince anyone that performance measurement isn’t a fad, or that it will be worth the investment. This book is for you only if you already believe this. This is not a book about theory or a regurgitation of existing KPI ideas. This book is about practical techniques, tested in the trenches.

This book is designed to help you fix your broken performance measurement process.

—Stacey Barr

1

What Exactly Is Performance Measurement?

‘What if I were to tell you that one of the most important keys to your organization’s success can be found in a very unlikely place—a place many of you consider to be complicated, inaccessible, and perhaps even downright boring? What if I were to tell you that this key to success is already one of the most ubiquitous and impactful forces in your organization? It’s there, waiting for you to tap into it.

This key to success is MEASUREMENT.

Measurement done right can transform your organization. It can not only show you where you are now, but can get you to wherever you want to go  .  .  .  measurement is fundamental to high performance, improvement, and, ultimately, success in business, or in any other area of human endeavour’.

—Dean R. Spitzer, Transforming Performance Measurement: Rethinking the Way We Measure and Drive Organizational Success

Terminology is inconsistent in the performance measurement and performance management fields, and it gets plenty of people confused before they even begin. To some, a ‘KPI’ is a ‘key performance indicator’, or the name given to the performance measures that are most strategically important to a company or organisation. For a company big on innovation, New Product Revenue could be a KPI. For a local government organisation trying to cope with drought, Domestic Water Consumption and Commercial Water Consumption might be KPIs. For others, a KPI doesn’t quantify amounts like the above examples but instead qualitatively describes an area of performance to be improved, like Enhance Employee Engagement or Grow Market Share.

Frustratingly, there isn’t a single universally accepted definition of any of the multiple terms used in the performance measurement field, which include ‘KPI’, ‘performance indicator’, ‘performance metric’, performance measure’, ‘lead indicator’, and ‘lag indicator’. Because there probably won’t ever be an accepted definition of any of these terms, let’s not waste our time worrying about their varying meanings.

I almost always use the term ‘performance measure’. To many, ‘KPI’ has become the friendly nickname for a performance measure, so depending on my audience I will sometimes use the term ‘KPI’. Getting my message out about how to do performance measurement properly is more important to me than finding and using perfect terminology. For me, understanding what a performance measure is matters more than what we call it.

What is a performance measure?

Too often, people treat measurement as a bureaucratic, jumping-through-the-hoops-of-the-planning-process activity. They come up with anything that can be written down in the performance measure column of their strategic or business plan that will avoid challenge from managers, executives, regulators, directors or the community. The result is usually a collection of measures that are the easiest, cheapest, most rudimentary information to produce. Often, these aren’t even true measures. And without realising it, managers who approach performance measures this way pay a price: such measures—or so-called measures—don’t give the feedback needed to inform management of the aspect of the organisation that they are in charge of monitoring.

People have widely varying views about what to put into their performance measure columns, but most of the things I’ve seen listed there do not qualify as performance measures. What I usually see instead of true performance measures are a combination of:

Initiatives chosen to improve performance

Events or milestones of the implementation of a strategy or project (supposedly chosen to improve performance)

Readily or easily gathered statistics that monitor the actions of the plan, and not the results the plan was designed to bring about

Sources of data such as survey or payroll system

Two or three vague words that could mean anything in general and nothing specific.

Initiatives are not performance measures

One challenge of using the word ‘measure’ is that it has two different meanings in the English language:

Meaning 1: We will take measures to improve customer loyalty.

Meaning 2: We will measure how much customer loyalty we have.

We could easily replace the word ‘measures’ with ‘initiatives’ in the first statement, but not the second. I’ve often wondered if this double meaning is the reason why so many people think that initiatives, projects, strategies, or ideas to improve performance are what belong in the performance measures column, next to their plan goals. Lists of performance measures generated based on this understanding read like the following:

Develop partnerships for the delivery of training with major universities

Appoint Head Coach and Team Manager early in the year

Secure philanthropic/Government funding for state-wide professional development program

Provide policy responses and submissions on key national issues relevant to rural and remote health workforce, education, and training

Identification and implementation of a measurement system to evaluate improved productivity, fruit quality, and reduced production costs for growers

These are not performance measures. Here’s why.

The first example, Develop partnerships for the delivery of training with major universities, was a performance measure for a goal that a college of surgeons defined as ‘Build effective partnerships with other educational bodies to improve delivery of educational material and enhance the training program’. Their performance measure describes what they have to do, not evidence of how well the goal is achieved, such as whether there is performance improvement in their training program.

The second example, Appoint Head Coach and Team Manager early in the year, was a performance measure for the following outcome that a sporting association hoped to achieve: ‘Increased capacity to send better-prepared senior and junior teams to multiple events’. Sure, getting a coach and manager on board early in the year is probably an important strategy for helping prepare teams, but simply appointing a Head Coach and Team Manager is not evidence of how much better prepared the teams are. This organisation would need to define exactly what they mean by ‘better prepared’ before they could identify a meaningful performance measure.

Initiatives are not performance measures in the same way that budgets are not profits and actions are not results. Yes, we need initiatives, actions, or strategies that we will implement to improve performance. But they are not evidence that our goal is reached. A university library might redesign the floor layout to provide more study space for students who are dissatisfied about the number of times they can’t find a space when they need it. The initiative would be a floor layout redesign project. The performance measure could be something like Average Student Satisfaction with the Ease of Finding Study Spaces, or Percentage of Students Entering the Library to Study Who Find a Study Space Immediately.

Any good strategic or operational plan will have goals, performance measures, targets and performance improvement initiatives, which should all be distinct from one another. The goal is the result that the organisation wants to achieve (which it currently hasn’t achieved). The performance measure tells us to what extent that result has been achieved over time. The target is the extent we want the performance measure to reach, which would signify achievement of the goal. Performance improvement strategies are the initiatives or projects that the organisation undertakes to shift current performance to the targeted level of performance.

Events or milestones are not performance measures

Strategic or operational plans commonly contain so-called performance measures that read like these:

Research, discussion, and position papers supported and/or written

Win Banksia Award for environmental achievement

Detailed Risk Management strategy adopted by the Committee of Management

Talent program implemented

Meet the Australian Medical Council (AMC) requirements for re-accreditation in 2011

Delivery of the agreed Environmental Works and Measures Program for each year of the Strategic Plan

Marketing campaign to educate the community developed for implementation in 2010

An organisation-wide skills audit is completed by Dec. 2010.

These are not performance measures. Let’s take a look at why not.

The first example, Research, discussion and position papers supported and/or written, was a performance measure for a brain injury association’s goal to ‘Encourage and conduct research and evaluation that is reliable, credible, valid, and relevant to individuals, families and communities’. Supporting and writing papers is not a performance measure because it fails to provide evidence of the result implied by the goal, which is that their research is seen as reliable, credible, valid, and relevant by their members. Having position papers supported or written is simply a milestone. By contrast, having those papers peer-reviewed and accepted for publication in a credible scientific journal might be evidence that they are meeting their goal.

Monitoring milestones is project management; monitoring results is performance management. Events or milestones fail to give us good performance feedback because they tell us only whether or not we implemented the actions that we’ve chosen to improve performance. They don’t tell us whether those actions worked and actually improved performance. Wouldn’t you agree that knowing what works and what doesn’t is fundamental to improving organisational performance?

Measures of activity completion are not performance measures

Another thing often confused with a performance measure is information about the means rather than the end result of a task. A measure of task activity or progress does not measure the performance result or the outcome that the task is intended to contribute to or achieve. Some examples of activity measures that are not performance measures:

Individual projects completed on time and at agreed cost

85% of staff have been briefed on the marketing plan

Delivery of the Routine Maintenance Program

The first example, Individual projects completed on time and at agreed cost, was an indicator listed for the following goal of a museum’s site improvements: ‘We aim to ensure that all of our resources are the best they can be’. In all honesty, this goal or key result area is very vague. We don’t know what defines ‘the best they can be’, so there is no way to meaningfully measure whether that goal is achieved. And regarding the proposed performance measure for this goal, on-time and on-budget completion of individual projects is only a single-point-in-time indicator of an activity having taken place. It does not tell us the how well that activity was performed. Completion of projects on time and on budget can be performance measures, but they’d be appropriate in a different context, for example a Program Office goal such as: ‘We finish projects on time and to budget’. This museum needs to be clear about what it means for resources to be ‘the best they can be’, and to measure aspects of about how good these resources are. Even that still sounds like a goal that’s way too broad: would we really want to measure how good all of our resources are? Aren’t there any high-priority resources that matter more than the rest?

Performance is about how well we achieve a result, not what we do to achieve that result. Direct evidence of a result is essential to properly test our hypotheses about how to achieve that result. If we have measures that track the means and not the end, then we probably need to have a team dialogue to fully describe what the end we’re aiming for looks like, and to design measures that provide evidence that end has been achieved. Activity measures can then be more useful because we will be able to correlate them with results.

Sources of data are not performance measures

Some so-called performance measures are really data collection processes, and not performance measures at all. Examples include:

Customer Survey

Occupational Health and Safety incident reports

Budget

SAP system

I’ve seen the first example, Customer Survey, in the performance measure column of many different organisations and company strategic plans, next to goals relating to improving customer satisfaction. How exactly does performing a customer survey mean that customer satisfaction will be improved? Of course a survey provides data, but which data specifically? And how exactly should those data be analysed to answer the question of how satisfied customers are?

These so-called measures are means of collecting data, but they do not provide the answers to our questions about performance. Performance measures can certainly be based on the data that these processes collect. But the measures need to be clearly designed and defined before the right data can be collected. When I worked in the Queensland Government Statistician’s Office as a survey statistician, I faced over and over the problem of data being collected without a clear idea of how they would be used. Our clients would first list all of the data they wanted to collect in a survey, without making it clear what they would use those data for or what business questions they needed to answer. Seeing data collection processes listed as performance measures is a symptom of not really knowing what the organisation’s actual performance measures are. Identifying a data collection process suggests that either we have only an inkling of what kind of data might relate to our goals or that we lack discipline in being clear and specific.

Measures should be designed before data is selected for the same reasons that a house should be designed before building materials are selected. If we start with the data, we risk ending up with the wrong measures and missing out on getting the data we really need for the right measures. Or we risk ending up arguing about exactly how to use the data to decide whether our goals are being achieved or not. Neither of these outcomes gives us a true measure of performance.

Do you have any data collection processes that assemble lots of the wrong kinds of data? Try to write down the business questions that you most need your data collection processes to answer, then identify the form that the answers to these questions should take. From there, you can identify the analysis that can produce these answers, and the data that this analysis would require. Once you’ve done this, you will be much closer to knowing what your performance measures really are (as well as the best data to collect to measure performance).

A few vague words don’t make a performance measure

These days we like to be concise and to the point—so much so, regrettably, that we have a great susceptibility to using empty words that in theory might sound very important and ‘key’ but in practice convey no meaning at all. We all know these kinds of words: ‘efficiency’, outcome’, ‘enhance’, ‘accountability’, ‘emergent’, ‘core,’ ‘effectiveness’, ‘holistic’, ‘key’, ‘reliability’, ‘quality’, ‘impact’, ‘sustainability’, and so on. Don Watson, author of Death Sentence: The Decay of Public Language¹, refers to these as ‘weasel words’, words that suck the meaning out of whatever is being stated similarly to how weasels apparently suck the contents out of eggs and leave the shells behind. Many, many performance measures use weasel words such as:

Media profile

Increased market acceptance

Research outcomes

The first example, Media Profile, was a performance measure for a strategic priority defined as ‘Reach out, engage, and lead on significant issues affecting multicultural and ethnic communities’ by a council representing ethnic communities. There isn’t a single defined and accepted definition of a ‘Media Profile’; this phrase is vague and open to interpretation. Users of such a vaguely worded measure will spend all their time arguing about what it means or what it should mean instead of making ground on their goal.

There are a few ways that organisations measure how much reach and engagement they have in the media. One example is Percentage of Positive Media Mentions, which would count up the number of media articles that show the organisation in a positive or neutral light (rather than a negative light) and take that count as a percentage of the total number of media articles. Another example is more qualitative; it has to do with the consistent image that the media repeatedly portrays of the organisation, and how well that image matches the way that the organisation wants to be portrayed.² For the measure Media profile, the organisation could count the number of media mentions, the number of lines or words published, or readers’ recall of, or responses to, media articles. This organisation needs to define specific measures that will indicate how they are portrayed in the media.

The second example, Increased market acceptance, was a performance measure for a strategy defined as ‘Work with the processing sector, listen to market signals’. This strategy of a livestock association was aimed at achieving the goal of ‘Be at the forefront of marketing and promotion’. What exactly is ‘market acceptance’? There isn’t a widely understood definition—certainly nothing as detailed as the definitions that exist for Customer Satisfaction Rating, Net Promoter Score, Earnings Before Interest and Tax (EBIT), Sales Revenue, Sales Conversion Rate or Staff Turnover.

The third example, Research outcomes, was the performance measure for a university’s goal to ‘Develop focused, relevant and world-class research’. What types of research outcomes are they looking for? They can’t know that until they define what constitutes ‘focused’, ‘relevant’, and ‘world-class’ research.

You don’t have a performance measure until you can describe how to calculate its values as a count, sum, percentage, average or ratio. Vague two- or three-word names do not make good performance measures. Every measure needs to have a name that makes it easy to refer to, but if that name is not widely and consistently understood we ought to accompany it with a short description that explains what the measure is. For example, a possible description for a livestock association’s measure called Market Acceptance might be ‘the percentage of consumer lamb purchases that are from our breed’.

Defining what a performance measure actually is

The so-called performance measures discussed above lack the features that are essential for a measure to provide good feedback about success or results.

One thing that often comes up when I talk about essential features of performance measures is the ‘SMART’ acronym. ‘SMART’ stands for ‘Specific, Measurable, Achievable, Relevant and Time-bound’. It’s a very good framework to test that our goals are well-formulated. But it’s most certainly not a framework for making sure our performance measures are well-formulated. Rather, well-formulated performance measures are the ‘M’ (‘Measurable’) element of SMART.

Well-formulated performance measures have these qualities:

A method of comparison that we can use to tell whether performance is good or not

A base of objective evidence that gives a reasonably accurate and reliable picture of current performance

A sufficient degree of granularity to detect small but important changes in performance to which we should respond

Relevance to the organisation’s priorities

The ability to show changes in performance levels over time, giving us enough context to avoid short-sightedness

Let’s look more closely at each of these five qualities.

A METHOD OF COMPARISON

A single number can never be a performance measure because there’s nothing to which we can compare it in order to decide whether the number is good or bad, on or off target, or in what direction it’s changing. We need to be able to compare actual performance with an ideal or desired level of performance, historic performance or, in the best of all possible worlds, both. If our Average Customer Satisfaction Rating measure is currently 5.6, how do we know whether that’s good or bad? How do we know whether we need to do something to improve customer service or not? We can know only if we compare that 5.6 to something else: a target rating, the rating scale itself, an industry benchmark of customer satisfaction for our sector, or historical trends in our Average Customer Satisfaction Rating.

A BASE OF OBJECTIVE EVIDENCE

It always frustrated me just how much time decision makers would spend during performance review meetings arguing about the data integrity of a measure rather than actually using the measure to help achieve a goal. Performance measures don’t need to have perfect integrity. They need only to be accurate and reliable enough to be trusted as information that will lead to a better decision than no decision or an uninformed decision.

The accuracy of a performance measure is akin to how close a shot comes to the bullseye on a target. That is, a performance measure is more accurate the more directly it measures the result we want to know about. Average Customer Satisfaction Rating is a more accurate measure of customer satisfaction than Number of Customer Complaints. An Average Customer Satisfaction Rating based on a random sample of all customers is more accurate than a volunteer sample because last thing we need is for our data to tell us only what the ‘squeaky wheels’ have to say, drowning out the valid, important and balancing views of the ‘well-oiled wheels’. Squeaky wheels, volunteer surveys, and easiest-ones-to-measure are examples of data sources that render our performance measures inaccurate.

Reliability is akin to how close the shots are to each other when a round of bullets hits a target. A measure is more reliable when it is based on enough data to make it precise enough, and when it continues to be precise enough as we measure it again and again over time. Would we rely on one day’s rainfall to draw conclusions about annual rainfall? What about five days’ rainfall? How many days’ rainfall would we need in order to get a precise enough estimate of annual rainfall? And what would this choice depend on? The sample needs to be large enough to accommodate the complexity of the result we’re measuring.

A SUFFICIENT DEGREE OF GRANULARITY

The way we design our performance measures can affect the granularity—or sensitivity to change—that those measures are capable of showing. Just think about two very common performance measures for customer satisfaction: the Percentage of Customers Satisfied and the Average Customer Satisfaction Rating. Both measures can come from the same set of data, which might be customers’ responses to a question about how satisfied they are on a 10-point rating scale. The Percentage of Customers Satisfied measure could be calculated as the percentage of customers who gave a rating higher than 5 on the 10-point scale. The Average Customer Satisfaction Rating is simply the average of all the ratings that customers gave. Imagine the situation where satisfied customers are becoming less satisfied and therefore rating their experience lower on the 10-point scale. The percentage measure would not pick up this shift nearly as well as the average measure because customers whose ratings dropped from 8 to 6 out of 10 would still be classified as ‘satisfied’. Measuring the degree of satisfaction rather than asking for a yes/no answer helps us pick up signals of change and therefore triggers us to take action sooner.

RELEVANCE TO ORGANISATIONAL PRIORITIES

We measure lots of things in business. But not all measures are performance measures, and not all measures deserve the same amount of our time and attention. Not all measures need to be acted upon either. Performance measures are a subset of measures that track the results related to our organisation’s biggest priorities right now. And the decision about which results are the biggest priorities is influenced mostly by the organisation’s strategic planning process, and partly by our knowledge about performance results that might not be in the strategic plan but are still mission critical or too important to ignore. Safety is a good example: we might not have a strategic goal to improve safety performance, but this is likely to be one of those performance results we should never ignore. Choosing the most relevant or important performance measures is a deliberate decision that we must make, because trying to measure everything would be a waste of time, money, and precious attention.

ABILITY TO SHOW CHANGE OVER TIME

One of the signs that an organisation or business has the wrong idea about the purpose of performance measurement is that they have annual targets and measures that are also tracked only annually. So how do they spend the year leading up to the target deadlines? They focus on doing the projects and initiatives they chose to hit those targets. But what if those projects and initiatives aren’t working? What if the organisation’s assumptions about how the world would unfold over the year were wrong? What if things have changed? If the organisation tracks their performance measures only annually, the management will find out the answers to these questions too late, at the end of the year when they calculate the measures for the year and compare them to the targets.

Unless we design measures that give us regular feedback over time, we’ll be faced with too little information too late. We won’t get the information that will help us fine-tune our strategies (the activities, initiatives, projects, and so on that are supposed to make performance better) to ensure that they actually produce the improved results they were supposed to produce. Performance measurement shouldn’t be a post mortem; it should be a health plan.

THE DEFINITION

So here’s a definition that brings all of these features into conscious awareness to help us recognise and create truly well-formulated performance measures that will directly serve the purpose of providing feedback for performance improvement:

A performance measure is a comparison that provides objective evidence of the degree to which a performance result is occurring over time.

To create performance measures that satisfy this definition, we need to be deliberate. It won’t happen without a deliberately designed approach. Brainstorming performance measures, shopping for off-the-shelf performance measures, hiring consultants to choose performance measures for us or building a performance dashboard are not true or complete performance measurement approaches.

What is performance measurement?

Performance measurement is a process, not an event. It is a series of steps in which we choose what to measure, define how we’ll calculate our measures, get the data, analyse the data to produce our performance measure values, graph and report our measures to highlight signals of performance changes, interpret these signals, and use this information to take meaningful action to improve organisational performance.

But performance measurement is not a stand-alone process. It is part of something bigger, something paramount not only to the improvement of organisational performance but also to the organisation’s very survival. Every organisation, just like every organism, has a renewal process: a process that seeks out faults, weaknesses or wear and tear, and then repairs, strengthens or regenerates itself. In living organisms, this renewal process usually responds to environmental factors such as a change in season or climate, physical demands, competition, threat, opportunity, or simply the passage of time. In the simplest terms, the renewal process has three parts:

The organism sets a goal for itself, such as running fast enough to catch prey.

The organism’s systems assess the gap between its current capability and the goal, using continual feedback about how well the organism performs relative to the goal (e.g., the level of muscle fatigue or soreness, cardiovascular fatigue, how often it catches prey, or how much energy it needs to expend in order to catch prey).

The organism’s systems close the gap between its current capability and the goal (e.g., by increasing the size of its muscle fibres, increasing the rate at which the lungs absorb oxygen, or modifying the strategies it uses to stalk the prey).

The renewal processes in our businesses and organisations are similar to those of living organisms. Without renewal, living organisms and organisations alike break down steadily and become less and less capable of surviving, let alone fulfilling their purpose.

An organisation’s renewal process has three parts, just like the renewal process of a living organism: the planning process to set goals, the performance measurement process to provide continual feedback about gaps between its current capability and the goal, and the improvement process to close the gaps. Let’s look at each of these parts of the process in more detail.

The planning process

For any organisation, the planning process involves setting the direction that the organisation is going to take. A planning process usually involves scanning the environment for trends or forces that are acting upon the organisation, understanding the needs or expectations of the organisation’s stakeholders, clarifying the organisation’s purpose, formulating a vision for the future, and looking at what capability the organisation currently has relative to what it needs to have to fulfil its mission and realise its vision. The planning process defines the gaps between where performance is, and where it should be.

Performance measurement plays a very important role in the planning process. Aside from giving us the means to monitor the size of performance gaps over time, measurement makes tangible the difference between where the organisation is now and its qualitatively articulated goals and objectives; that is, measurement makes tangible the gaps between ‘as-is’ and ‘to-be’ performance. When strategy is made more tangible through performance measures, something magical happens to everyone’s understanding. Cynicism is replaced by curiosity, complacency by commitment, and mediocrity by a passion to lift the bar.

The improvement process

In the improvement process, decisions are made and actions are taken regarding how resources will be applied to effect change. The change might be to acquire new equipment or infrastructure, new skills or knowledge, new practices or policies, or more people. The purpose of the improvement process is to close the gaps between as-is and to-be performance.

The role of performance measures in the improvement process is to focus attention on what most needs improving, and to guide the improvement efforts by testing which actions are causing as-is performance to move closer to to-be performance and which aren’t.

The performance measurement process

The performance measurement process is effectively the feedback loop in the organisation’s renewal process, providing the communication link between planning and improvement so that strategy can be implemented. The performance measurement process provides vital information about the organisation’s current levels of performance in relation to the direction defined in the planning process. That information is then captured, analysed, and communicated so it can inform the improvement process. Another way to say it is that the performance measurement process monitors the size of the gaps between as-is and to-be performance.

One of the very common reasons that performance measurement processes break down is that people just don’t have the big-picture perspective of why we measure performance. Who in your organisation understands what performance measurement is truly about? Don’t you think everyone should?

As I mentioned earlier, our performance measurement processes need to be deliberately designed if we’re going to have meaningful performance measures. This won’t happen serendipitously, or by continuing to do the same things we’ve always done to measure performance. If we’re measuring anything at all in our organisations right now, chances are that our performance measurement process is somewhat ad hoc and unstructured. Unless we have well-formulated performance measures throughout our organisation or business, and those measures are routinely hitting targets and making performance fundamentally better, then our performance measurement processes need to be improved.

Dealing with the terminology problem

As you will recall from the start of this chapter, words frequently used in the performance management field include ‘performance measure’, ‘metric’, ‘performance indicator’, ‘key performance indicator (KPI)’, ‘key result indicator (KRI)’, ‘lead indicator’, ‘lag indicator’, ‘initiative’, ’strategy’, ‘goal’, ‘objective’, ‘target’, ‘priority’, ‘critical success factor (CSF)’, ‘key result area (KRA)’, ‘strategic theme’, ‘vision’, ‘mission’, and no doubt many more.

The problem isn’t the sheer number of words we use. The problem is that we use them differently, with varying and overlapping meanings, and fail to explain our meanings to each other. For example, here are some variations in definition:

David Parmenter, author of Key Performance Indicators: Developing, Implementing and Using Winning KPIs, defines some of these terms like this:

A key result indicator (KRI) tells us ‘how you have done in a perspective’

A performance indicator (PI) tells us ‘what to do’

A key performance indicator (KPI) is a performance measure that tells us ‘what to do to increase performance dramatically’, as opposed to other types of performance measures.

By contrast, the KPI Library (www.kpilibrary.com) uses the term ‘KPI’ to refer to any kind of measure at all that a business or organisation might use to monitor its performance. They don’t make the distinctions that Parmenter does.

And others define these terms differently, perhaps not in a book but certainly by the way they use the terms. I’ve often seen people using the term ‘KPI’ to mean a qualitative goal, rather than a quantitative measure. They will say things like, ‘We must measure our KPIs!’

And remember, ‘KPI’ is just one of the many performance management terms that does not have a standardised, universally accepted definition. I have no idea how this problem of varying terminology is going to be resolved (and that’s not the intent of this book), but I raise the issue again because I am going to use terms in this book for which you may have different meanings. So let me tell you my definition of these terms, and then you can map your own terms to my meanings and thus avoid distraction and confusion as you continue reading.

Take, as our starting point, the following plan that describes part of the strategic direction for Chokolaide, a Swiss not-for-profit organisation that supports people with a lifestyle-diminishing addiction to chocolate.³ For now, I’ve labelled each component of the plan as Object A, Object B, etc.

You probably have your own names for the ‘Objects’ used in this fairly typical plan layout. Here are the words I use for these objects, in this book and in my work:

Object A—vision statement

Object B—mission statement

Object C—organisational values

Object D—performance result

Object E—performance measure

Object F—target

Object G—strategic theme

Object H—improvement initiative

Feel free to substitute your own words, but make sure you use them consistently, and always be prepared to explain the meaning of the words you choose. And in reading this book, keep in mind the definitions established here.

What is the PuMP Performance Measure Blueprint?

PuMP was inspired by the struggles I had in my first Measurement Consultant job at Queensland Rail, a government-owned corporation that was the only remaining fully-integrated railway in Australia at the time. Queensland Rail encompassed urban passenger trains, long-distance passenger trains, small freight, bulk freight, coal and minerals freight, infrastructure management and rolling stock maintenance—the whole kit and caboodle when it comes to railways. It was a rich, varied organisation in which to learn, explore and experiment with ways to meaningfully measure performance, and to experience all the technical and social obstacles that get in the way of measuring performance.

My role as Measurement Consultant was a challenge. I often felt quite lost and alone because there were no performance measurement frameworks or methodologies to follow, and I had no measurement mentor to teach me the ropes. People weren’t excited about measurement; they wanted to avoid it as much as possible. Bad practices in data collection, analysis, reporting, and interpretation of quantitative information abounded, and hampered much of the effort to improve the selection and use of performance measures. Working in that environment was a huge contrast to the certainty I had been used to in my previous jobs.

Before I joined the railways, I was a research statistician with post-graduate qualifications in mathematical statistics. This grounding in the scientific method couldn’t help but influence my ideas about measuring organisational performance. With these qualifications, I started off stronger on the science and technical side—and almost oblivious to the social and political complexities of measurement. But my great manager, Kevin, supported, encouraged and helped me develop to become strong in the social aspects of the job too.

One of the starkest lessons I had in the social aspects of performance measurement was the day I showed a fabulous time-series graph of a safety measure (I can’t remember which one; it might have been train derailments, or trains running through red signals) at a safety committee meeting. The safety manager took one look at the graph and literally yelled at me that I had no idea what I was doing, and that the graph was useless. Actually, the graph was saying that he had managed to achieve absolutely no improvement in that area of safety performance during the previous two years. While I was mopping up tears in the ladies’ bathroom and feeling a bit sorry for myself, I realised that I hadn’t actually shown a graph that would enable him to recognise or think about how to improve the problem. Instead, I’d shown the rest of the safety committee a graph revealing that he was incompetent. Ouch. However good the quality of the information I presented, it was not likely to be well received or used without buy-in from the person responsible for the problem.

Lots of experiences like this one forced me to find ways to do my job better, both on the technical side and on the social side. Because there was no performance measurement methodology to guide me, I invented one that reflected what I learned. Devising a performance measurement methodology wasn’t what I set out to do; it just evolved as I met new struggles and tried different ways to overcome them. As a result of my own struggles, I worked on making data collection easier, on strategies that would be faster than brainstorming to find meaningful measures, and on getting people interested in measuring. The methodology that resulted from these years of struggling, testing, and tuning is PuMP.

I didn’t call it PuMP in the beginning. I called it the ‘Performance Measurement Process’ because to me the key to measuring performance easily, meaningfully, and engagingly was to recognise that it is a process, not a discrete event. I kept calling my methodology the Performance Measurement Process until one day a client blurted out, ‘You should call it PuMP!’ I asked her what she meant. She explained that she wanted a catchy, shorthand way to refer to the methodology. When I asked her what the ‘u’ in ‘PuMP’ stood for, she replied, ‘Because good performance measurement is up to u!’ Including the ‘u’ was just a cute way to make a word out of an otherwise boring acronym. But it stuck.

The steps in the PuMP approach to performance measurement are both technical and social, and that’s why it creates good measures, good measurement processes, good performance management and a strong performance culture. If we want a collection of meaningful and engaging performance measures that align to our strategy, and make it easier to execute that strategy and improve organisational or company performance, then certain things have to happen. These things are both technical and social, and they weave together to make up our performance measurement process.

The technical things are to do with:

How we articulate our strategy so that it’s measurable

How we choose performance measures that meaningfully align with that strategy

How we implement those performance measures, from data collection to analysis and reporting

How we interpret and use the measures to successfully improve organisational performance.

The social things are to do with:

How we frame the purpose of performance measurement to people who are cynical and sceptical about it

How we engage people in choosing the performance measures so they feel ownership of them

How we bring performance measures to life without people whining about how it’s taking their time away from ‘real work’

How we encourage people to use performance measures to take actions that improve organisational performance, as opposed to knee-jerk reacting to every change every month, or just making the numbers look good.

These technical and social dimensions of performance measurement come together into the PuMP Performance Measure Blueprint. Implementing the Blueprint is a team-based process, with the eight deliberate steps outlined in Figure 1.

Step 1: Understanding Measurement’s Purpose After we form our Measures Team and take a few other preliminary steps described in Chapter 3, the Blueprint begins with using a self-assessment tool called the PuMP Diagnostic. The Bluprint helps the team understand what good measurement means and how to do it properly, while avoiding the most common struggles and problems. The diagnostic step is covered in detail in Chapter 4.

Step 2: Mapping Measurable Results In Step 2, the Measures Team explores why it’s so hard to measure the right things; how to find out whether their strategy (i.e. goals or objectives) is measurable enough; and how the two templates called PuMP Measurability Tests and PuMP Results Map can make the team’s goals or objectives clear, specific, and easily measurable. This step is covered in detail in Chapter 5.

Step 3: Designing Meaningful Measures In Step 3, the Measures Team learns to recognise the traditional approaches that people take to selecting measures, and why those approaches fail. They then use the PuMP Measure Design template to design measures that provide evidence of their goals, that are meaningful, relevant, and cost effective, and that will make it easy for people to buy in. This step is covered in detail in Chapter 6.

FIGURE 1. The PuMP Performance Measure Blueprint
FIGURE 1. The PuMP Performance Measure Blueprint

Step 4: Building Buy-in to Measures To avoid the risk of their colleagues and managers not buying in to performance measures they create, the Measures Team uses the PuMP Measure Gallery technique to build buy-in and generate excitement, both for performance measurement in general and for their new measures in particular. This step is covered in detail in Chapter 7.

Step 5: Implementing Measures In this step, the Measures Team learns why so many measures are never successfully brought to life (i.e. never reported or used). They use the PuMP Measure Definition technique to detail exactly how each of their measures should be implemented, and begin the documentation of a Corporate Performance Measure Dictionary. This step is covered in detail in Chapter 8.

Step 6: Reporting Performance Measures In Step 6, the Measures Team learns why typical performance reports are next to useless, even when these reports employ the latest and greatest dashboard dials, gauges and graphs. They use the PuMP Report Design technique to quickly and easily produce performance reports that answer the three most important questions that every performance report should answer, which are covered (along with the other details of this step) in Chapter 8.

Step 7: Interpreting Signals from Measures In this step, the Measures Team discovers why it’s so easy to misinterpret performance measure signals and how, frighteningly, most people do misinterpret them. The team then follows the PuMP Interpreting Measures technique to make the true signals of change in their performance measures easier to see, statistically valid, and incredibly useful for producing insight. This step is covered in detail in Chapter 9.

Step 8: Reaching Performance Targets The role of performance measures is to facilitate quick, easy and economical performance improvement. The PuMP Using Measures technique enables the performance team to make substantial improvements, moving performance closer to meaningful targets. This step is covered in detail in Chapter 10.

The eight activities in the PuMP Blueprint describe a process that brings performance measures to life. The eight steps are generic activities that we need to perform regardless of which strategy or performance framework we prefer—the Balanced Scorecard, the Performance Prism, Triple Bottom Line, the Baldrige Criteria or the Six Sigma Scorecard. These kinds of performance and strategy frameworks are often a good input to Step 2 in the PuMP Blueprint, which is about deciding which aspects of our strategy we should measure. After all, we need to know what matters most before we can find measures that are meaningful. But most of these frameworks still leave us wondering how exactly to go about finding the most meaningful performance measures. (Some even suggest you ‘brainstorm some measures’—Aaargh. Don’t get me started on the problems with the vague directive to brainstorm measures, or the hodge-podge that results from such a vaguely guided brainstorming activity. Moreover, these other performance and strategy frameworks rarely acknowledge any of the other activities that are essential to meaningfully measuring organisational performance, which the other steps in the PuMP Blueprint address.

If any of the eight PuMP Blueprint steps is missing, or not performed effectively in our performance measurement process, we’re taking big risks. Without thinking carefully about which measures to select, we risk having measures that aren’t relevant to our purpose, or don’t help us understand the causes of current performance results. We also risk measuring too much. And because measures are such strong attention magnets, measuring the wrong things (or too many things) will take attention away from what matters most. Do you have measures that aren’t meaningful, not used, or not driving performance improvement? PuMP solves those problems for you.

Questions for a quick self-assessment

Let’s do a quick self-assessment of whether your organisation has fallen into any of the pitfalls of poorly or incompletely designed performance measurement approaches. Ask yourself the following questions:

How do you feel about the quantity and quality of the data your organisation has available for the measures that truly matter? Without deliberately designed data collection and capture systems, the performance data from which your measures are computed can be biased, inaccurate, imprecise, too costly or too cumbersome for people to treat with consistent care. Or the performance data is never collected at all, and important measures are never brought to life.

Are you finding that performance isn’t improving consistently or fast enough to reach your targets as planned? Inappropriate analysis and reporting lead to misinterpretation of the messages the data really contains, so you won’t be interpreting your performance measures well enough to get all the important information it can offer. You’ll react to trends that aren’t really there, and miss important trends that are there because you just can’t see them.

Is most of the time in your performance review meetings spent debating data integrity or measure relevance, resulting in decisions that are made based on what, if you’re honest, amount to whims, gut feelings, and hearsay? Poor presentation can inadvertently misinform, mislead and complicate decision making, resulting in more time spent debating the information than exploring root causes and identifying potential solutions.

Are people fudging the figures rather than improving process performance in your organisation? Unless you think about how performance measures should be interpreted and acted upon, you risk driving the wrong kinds of staff and manager behaviour, encouraging them to manage the numbers rather than the outcomes.

How many of the eight activities in the PuMP Blueprint does your organisation carry out in a deliberate and systematic way? Do you know which departments and people contribute to these activities, or do they somehow just get done? Do you have a good idea about the kinds of resources needed to properly perform these activities? Does your organisation appreciate how much time and effort are involved in performing these activities properly? Do people have the right skills to perform these activities, or are your staff and managers guessing or doing what has always been done? Is there a team or formal structure for managing the performance measurement process, or is the process ad hoc and inconsistent across your organisation?

Unless your organisation already has processes, skills and resources committed to all of the steps in the performance measurement process, your current frameworks for selecting measures probably haven’t gotten you any further than having a document that lists (and maybe defines) your ‘Key Performance Indicators’. This is hardly enough of a return on all the time, effort and money you’ve invested (and possibly wasted) to obtain powerful performance information that will help you manage and improve your organisation’s performance. But don’t throw the towel in just yet. If you have made it this far, then what you have now is a foundation for improvement of your organisation’s performance measurement process.

How the other performance measurement methodologies fit

Many of the performance measurement methodologies on the market today aren’t actually performance measurement methodologies at all. They might be framed as performance measurement methods because they help give us a framework for deciding what to measure, or because they espouse the importance of measuring performance. But they are mostly strategy design methodologies. They almost always lack sufficient detail to help us choose really great measures and bring those measures to life in performance improvement decision making.

The most common of these methodologies are the Balanced Scorecard, the Performance Prism, and the various program logic or outcome models. In a minute, we’ll take a closer look at these methodologies and what they do and don’t offer. But we don’t have to give up any of these frameworks if we adopt the PuMP Blueprint. In fact, we still need these other methods because we’ll have meaningful performance measures only if we first know which performance results are important to measure. And the strength of these other frameworks lies in their ability to help us make a clear, deliberate decision about what matters most for our organisation’s success—that is, what our strategy will be.

Balanced Scorecard

The Balanced Scorecard, the brainchild of Drs. Kaplan and Norton and conceived in the early 1990s, is less of a measurement tool and more of a powerful strategy tool. Not many people appreciate this, and the word ’scorecard’ featured in this framework’s name is no doubt a big reason why.

The Balanced Scorecard is a powerful strategy tool because it asks us to be very deliberate about what truly matters for organisational or company success. I’ve become an advocate of the Balanced Scorecard Institute’s ‘Nine Steps to Success’⁴ methodology because it is the most deliberate approach I’ve come across for designing a strategy that is focused and measurable.

So why isn’t the Balanced Scorecard a measurement tool? The emphasis of the Balanced Scorecard literature is designing a strategy to encompass the four Balanced Scorecard elements: financial performance, customer performance, internal business process performance, and learning and growth performance.

Selecting performance measures is certainly a part of implementing a great Balanced Scorecard. But little in the Balanced Scorecard literature provides detailed advice on designing or finding truly meaningful measures for these four elements of organisational performance, or on implementing measures. When instructions say ‘Now brainstorm some KPIs’, or give us a list of performance measures to use, we’re in trouble. If we have a goal that isn’t covered by examples provided in books or case studies, where do we look for performance measures?

Another tool related to the Balanced Scorecard is Strategy Mapping. This tool, as its name suggests, is about strategy design, not performance measurement. Yes, we will measure what’s in our strategy map, but how exactly do we find the right performance measures for our unique strategies? Strategy Mapping does not answer—and was not designed to answer—this critical question.

Performance Prism

Not unlike the Balanced Scorecard, the Performance Prism⁵ is a framework for identifying the important things for us to focus on to maximise organisational performance and success. Where the Balanced Scorecard focuses on the shareholder view as the ultimate outcome for companies (or the customer view for non-profit organisations), the Performance Prism takes a multi-stakeholder view, including shareholders, customers, employees, suppliers/partners, and regulatory and other communities. The Performance Prism’s perspectives start with stakeholder satisfaction and contribution, and also include strategies, processes and capabilities.

The Performance Prism is more measure-focused than the Balanced Scorecard as far as offering a process for developing performance measures. Its guidelines are: design, plan, and build; implement and operate; and then refresh. But this process starts with defining the details of our performance measures, not with how to design or choose those measures in the first place.

I once attended a workshop about the Performance Prism, and we were learning the method for defining measures (similar to Step 5 in the PuMP Blueprint). In groups, we undertook an activity to define a measure for innovation. The group I was in spent the entire time debating what a good measure would be. We had no process for designing or choosing a measure of innovation, and therefore we were not able to define any details at all using the template we were given.

Program Logic Model

Some governments and other organisations not driven by profit use versions of program logic or outcomes models. Program logic models visually map the cause-effect relationships that an organisation understands to exist among the inputs to its programs, the outputs produced by those programs, and the outcomes these programs have for targeted stakeholders. Program logic models also provide a framework for assessing the impact achieved by the organisation’s application of resources to its programs. These models are intended for organisations whose impact is social change, such as reducing health problems from smoking, reducing water consumption in times of drought, increasing use of sunscreen to minimise skin cancer incidence, or reducing homelessness. The input-output-outcome thinking that program logic frameworks encourage is very helpful in keeping performance measures focused on what matters, and I would suggest this thinking is useful beyond just social change organisations.

Like the other models we’ve looked at, most of the program logic literature focuses on how to build a program logic model, not on the nitty-gritty steps to finding and implementing the appropriate performance measures for our specific model.

Performance measurement is so much more than ‘selecting some indicators that align with our strategy’. It’s about finding the best performance measures that provide evidence of our unique goals. It’s about designing those measures if we can’t find any existing ones that suit our organisation. It’s about bringing those measures to life in a way that engages people to monitor performance regularly. It’s about responding only to true signals revealed by the measures, and to use the measures to continually learn how to improve the organisation’s or company’s capability to fulfil its mission and achieve its vision.

But most performance measurement processes aren’t capable of doing this. And it will take a little work to help our colleagues understand this so that we can enjoy their support as we tackle an overhaul of our performance measurement process using the PuMP Blueprint.

Take Action: Describe your current performance measurement process

How do performance measures get selected, implemented and used in your organisation? Who gets involved? What timeframes are typical for the different stages of choosing and reporting measures, and using them to pursue targets? Make a flowchart showing how this happens in your organisation. If this process happens differently each time, document that. If it happens differently in different parts of your organisation, document that too. The more ad hoc and inconsistent your measurement process is, the harder it will be to document it clearly and succinctly. But do your best, so you can begin to see and appreciate where your organisation’s performance measurement approach is now.

Chapter summary

A performance measure is a comparison that provides objective evidence of the degree to which a performance result is occurring over time. There are different names for performance measures with different names including ‘KPIs’, ‘lead indicators’, ‘lag indicators’, and ‘performance indicators’, but there are no universally accepted definitions of these terms. In this book, we use the term ‘performance measures’, or occasionally ‘KPIs’.

The primary reason that we measure performance is to support organisational renewal. Measurement is a feedback process more than an assessment process. Performance measurement shouldn’t be a post mortem; it should be a health plan.

Performance measurement is a process that produces performance measures, ready for decision making. It involves choosing measures and defining the details of how to calculate them, report them, interpret them and put them to use.

The PuMP Performance Measure Blueprint is a series of eight activities that guide us to design and implement our performance measurement process so that we create meaningful measures that support performance improvement and a strong performance culture. The PuMP Blueprint works with other performance and strategy frameworks such as the Balanced Scorecard and the Performance Prism, as well as business excellence models such as EFQM, Baldrige and the ABEF. It provides the detail to meaningfully measure the strategy that we create with these other frameworks.

Now, let’s take a closer look at our performance measurement processes.

2

Does Your Performance
Measurement Process Work?

“If you can’t describe what you are doing as a process, you don’t know what you’re doing.”

—W. Edwards Deming

“Revolutions in science have often been preceded by revolutions in measurement.”

—Sinan Aral, professor at the New York University Stern School of Business

For more than a decade, my clients and readers have been telling me about all the things that go wrong with performance measurement at their organisations. And the same problems come up time and again in everyone’s stories. So before we dive into the details of the PuMP process in Chapter 4, let’s look in this chapter at some of the common ways that people struggle with performance measurement. I’ll show you the symptoms of a broken performance measurement process, as well as a preliminary activity to warm people in an organisation up to the PuMP process and how it can help resolve the struggles and fix the performance measurement process.

The common struggles with performance measurement

Many of the problems with performance measurement happen right at the start of the process, before any data are even collected or reported. In other words, lots of performance measurement problems start before we even try to look for measures.

Not prioritising leads to too many measures

I knew a manager who was passionate about business excellence and continuous improvement. Very passionate. He valued people, he understood his business processes, he believed in pursuing excellence, and he had a strong work ethic. I really liked him (and still do). His trouble was that he didn’t prioritise. In the strategic plan for the business, he had a total of 87 KPIs because he believed that everything mattered and that his managers and staff needed to give priority to everything. And as he drove them to do just that, they burned out with exhaustion. Meanwhile, he literally had a nervous breakdown himself. He’s a lot different now, in no small part because he’s a great learner. Today he’d agree with you and me that nothing stands out as ‘key’ when you have 87 KPIs. We achieve more and better when we focus on fewer priorities. The strategic plans that guide us need to focus us on a small number of the most important goals. And in our performance measurement process we should measure only what matters for pursuing and achieving those goals, with an eye on reviewing our goals regularly to ensure that they are not producing unintended consequences. I don’t go for the idea of pursuing goals with blinkers on and hitting targets just because we set them. Goals need regular review to make sure that the benefits of achieving them will result in an outcome consistent with our intentions and desires, without causing harm as an unintended consequence. We’ll talk more about unintended consequences later in the book.

Using brainstorming leads to meaningless measures

Another very common problem is to mistake brainstorming for a performance measure selection technique. It’s very easy to see when an organisation or business has brainstormed their performance measures because certain types of performance measures—or at least things that people think are performance measures—will dominate what’s listed in the performance measure column of the strategic or business plan. Using a local government and the strategic plan goal of reducing waste in the community as our example, the types of performance measures that typically result from a brainstorming session might include: ‘Implement waste management strategy’ and ‘Community adoption of improved waste practices’. These aren’t performance measures at all. They fail to focus people on results, only inform us only when it’s too late to change anything, and more often than not are never even implemented because no one knows how to implement them. This is but one of many examples demonstrating how people get confused about what measures, targets, actions and KPIs are, and how these things are distinct from one another.

Ad hoc reporting leads to waste and duplication

Reporting of measures is another potential problem, and an area that can lead to tremendous waste. Here’s an anecdote illustrating the problem: A railway corporation’s coal division had so many ad hoc and independently produced performance reports making their way to the general manager that he felt more confused than enlightened. For just one very important performance measure, Average Cycle Time of Trains, the manager struggled to figure out why he had 12 different versions of this measure, and which one was telling him the truth. The reason for the variety of measures was that business analysts were using their own local definitions of Average Cycle Time of Trains. Some were using different start or end times. Some were leaving out trains that ran more than three hours late. Some were adopting the customers’ definitions of the measures. Some were drawing data from train operations systems, and others from manually updated Excel spreadsheets. The general manager was so frustrated with the number of divergent reports, and how little insight he was getting from them, that he joked about printing a copy of every report that he got in a month and loading the copies into a wheelbarrow to push into the next management meeting to make his problem very tangibly understood. Nothing was going to change until there was some systematic control over the independent performance reporting industries that were flourishing all around the coal division. The change required the general manager to decide exactly which measures he needed, and exactly what was the correct way to calculate each of them, to provide evidence of the performance results that mattered most.

Traditional reporting leads to misreading signals in measures

Even when measures are successfully implemented, problems can arise in the communication of the measurement results to the people who use them. Although I am a great fan of information dashboard technology for its power to rapidly and professionally display performance measures, one of my pet peeves is just how poorly the power of dashboards is understood and applied. I heartily endorse visual analysis expert Stephen Few’s well-founded and superbly articulated critiques of the inappropriate uses of gauges, dials, and particular chart types, colours and other design features that are everywhere in the world of dashboards (in particular, see his book Information Dashboard Design⁶). Very importantly, dashboards often fail to highlight the real signals in performance measures.

Although dashboard applications can apply ‘traffic lights’ that turn red, amber, or green to draw our attention to particular measures on the dashboard, the rules used to determine which measures are red, amber, or green are often statistically invalid. These rules are typically based on comparisons of this month to last month, or this month to a target, with standard tolerances built in for what constitutes a red, amber or green variation between months. If this month is more than 10% worse than last month, the measure will be flagged by a big red cross (or a gauge needle pointing into the red zone). But how do we know that 10% isn’t just the normal amount by which this measure naturally varies? Different measures have different amounts of inherent variability. (How often do we see a performance measure that’s the same every month, without fail?) Variation is natural, and not all variation is a signal that something needs attention. True signals are changes or deviations from the pattern of natural variability. And we need more than one data point—more than one month’s performance—to assess this. A measure of System Outages, for example, shows a value of 42 in May 2006 and a value of 26 in June 2006. To the statistically illiterate, this is a 38% improvement that would get a green light on the dashboard or in the monthly report. But when we look at System Outages as a time series in Figure 2, we see that the 38% difference between May and June appears to be a typical amount of variation for this measure. In fact, we can see that System Outages actually seem to be increasing, in general, and not improving at all.

Figure 2. System Outages: Getting better, or getting worse?
FIGURE 2. System Outages: Getting better, or getting worse?

If you think that perhaps the positive difference between May and June in 2006 in Figure 2 was a sign of improvement, think again. Figure 3 shows that System Outages continue to vary widely, with no sign of the average level coming down.

The charts that we see in Figure 2 and Figure 3 are XmR charts. Chapter 10 explains the XmR chart’s purpose and value, and gives instructions for using them. But for now it’s enough to know that the horizontal flat line in these charts (known as the ‘central line’) is a gauge for the overall average level of performance of the measure System Outages. The shaded band, which indicates what are known as the ‘natural process limits’, shows the amount of inherent and routine variability in the performance of the measure System Outages. Neither the central line nor the natural process limit is a target or a standard. That is, they are telling what current performance is, not what it should be. An XmR chart is incalculably superior for flagging true signals in performance measures compared to the month-to-month, month-to-target, month-to-year-to-date, or year-to-year comparisons that underlie most information dashboards’ red-amber-green alerts.

Figure 3. System Outages is getting worse
FIGURE 3. System Outages is getting worse

Often people don’t see that there’s a problem in their measurement process staring them right in the face and profoundly affecting the quality of their decision making. Here’s an example: the senior management team in an energy utility was drawing the wrong conclusions from one of their measures of the reliability of the electricity supply, a very important measure to both customers and industry regulators. The reason they were drawing the wrong conclusions was that the way the measure was presented masked a major problem. The measure was shown in a line graph as a 12-month moving average. This means that each point on the graph was calculated as the average of the previous 12 months’ values of reliability. The graph showed a steady and gradual downward trend over the previous 18 months—a good result because the graph measured electricity supply problems that management wanted to reduce. This graph gave senior management the impression that the problems were decreasing, and reliability was improving.

But then I gave them a second graph that showed the actual reliability in each month on the graph instead of the moving average. This graph showed a very different pattern from the first one. Rather than an improvement trend, the actual reliability graph showed a step change when, 18 months previously, the average reliability suddenly dropped and immediately stabilised at a new lower level. The leadership team was surprised to learn that the sudden drop in reliability problems was due to an industry-wide standardisation of the definition of the reliability measure. So while they had believed that reliability was improving, nothing was changing at all.

Misinterpreting or hiding the true signals in performance measures and their causes is often unintentional. This is a very important lesson that the majority of users of performance measures need to learn.

Using measures to monitor people leads to fudging the figures

Another problem that often happens with performance measurement is one that no doubt we’ve all come across in our own travels. It is probably the most talked-about problem: when people fudge the figures to make the measure look good while actual performance erodes away. Call centres are a classic setting where this problem manifests because of the way performance is often measured there. One of the measures typically used in call centres is the Average Handling Time, which is the sum of two other measures, Talk Time and Post-Call Processing Time. Talk Time is the average time, in seconds, that call centre operators spend talking with customers. For the call centre as a whole, this is a useful measure to understand caching opportunities. And average call duration is helpful for forecasting things like how many staff might be needed in the future. But there’s a problem with how this measure is usually recorded. A common practice is to measure the Talk Time for each individual customer service representative (CSR). This is not good because it focuses the CSR on the seconds they spend talking to the customer rather than the quality of the conversation. When we add to the CSR’s performance measures the measure Post-Call Processing, which is the average time in seconds that it takes the CSR to finalise any transactions related to the call after the customer has hung up, we create another problem. If you have ever waited on the phone while a CSR processed your request, you would probably find that the CSR’s performance was measured in relation to a time target for Post-Call Processing. To reduce the time spent on Post-Call Processing, the CSR kept you on the line while they did some of the processing even though the processing did not require your input or provide you with any value. The CSR wasted your time because they were simply trying to hit the call centre’s Talk Time and Post-Call Processing targets. Because the CSR is not measured on the customer’s time investment or satisfaction, they do not worry about either of those factors. We know the lesson of this tale already: what we measure is what we get.

Focusing on our circle of control leads to excuses and blame

And finally, one of the most tragic of performance measurement problems: when a measure is well chosen, efficiently implemented, clearly reported, owned, and valued, but still no performance improvement comes of it. It’s not hard to imagine that revenue would be one of the important performance measures for a commercial company with a growth strategy. This was the case for a vocational education institute. Revenue was monitored monthly, and discussed at each regular performance review meeting. Each meeting was the same: the senior leadership team would review the revenue graph and note that it was steadily decreasing, rather than gradually increasing toward their target. Each meeting the conversation was the same: ‘The economy is tough right now’, ‘Other education institutes are also suffering revenue decline’, ‘We can’t control the economy, so we’ll just have to ride it out’. I’ve seen this pattern often, and heard about it from my clients often, too. Performance management and performance improvement aren’t easy, and the solutions to close the gaps between actual performance and targeted performance rarely lie within our circle of control. But often they do lie within our circle of influence. Many decision makers don’t look there though, and are quick to blame outside circumstances and find excuses to do little or nothing.

Most of the causes of all these performance measurement problems can be traced back to the design of our performance measurement processes. Let’s look at some of the signs that tell us our performance measurement processes need repair.

The symptoms of a broken performance measurement process

The ultimate reason for measuring performance is to help us improve performance. Measuring performance allows us to improve it faster, more easily, and by larger leaps than if we don’t measure it because measuring provides feedback and focus far sharper than our gut feelings, experience, or even observations with our own eyes can ever give us. Well-designed measurements give us accurate, timely information about how things are working. As a result, we are much more likely to hit the bullseye with our decisions and actions to make things work better. As an organisation gets better at performance measurement, performance improvement happens faster and more easily, and the organisation accelerates toward performance excellence, fulfilling its mission and achieving its vision.

But performance improvements need to be targeted at critical areas. An organisation can do a superlative job of improving the performance of all kinds of things that have little to do with its mission, vision, or strategic priorities. However, meaningful performance improvement is about continually closing the gaps between actual and targeted performance in areas that are deliberately selected as strategically important or mission critical. Performance measurement excellence means improving performance in this way consistently, across the whole organisation, as a natural part of how the organisation operates.

How frequently important performance gaps are being closed, and how extensively this is happening throughout the organisation, depends to a large extent on the quality of the organisation’s performance measurement process. There are many, many symptoms of a broken performance measurement process. I don’t want to dwell too long on this topic, as I’m sure we’re all keen to get started on fixing our performance measurement processes rather than lamenting over them. But it will help to understand the baseline from which our performance measurement processes are going to be improved. Assessing the problems in an organisation’s performance measurement process also makes for a refreshing discussion among leaders, colleagues and staff coming together to identify what must change to achieve performance measurement excellence. So here’s a sample of some common symptoms of a broken performance measurement process. Take note of which ones you recognise in your organisation:

People don’t appreciate the need to measure, or to have better measures.

There are no goals, objectives or results that define what is important to measure.

The focus is more strongly on financial measures than on non-financial measures.

People have given up on measuring seemingly immeasurable, intangible goals.

There are too many performance measures.

Existing measures don’t align with strategic direction, team goals or purpose.

Milestones and initiatives are used as measures.

Measures are named so vaguely that it’s not clear exactly what they are or what they mean.

There is little awareness of performance measurement among people who aren’t directly involved in choosing, implementing and using the measures.

People complain that measuring takes too much time and effort.

There is little buy-in or engagement from people to measure performance (they say they have ‘real work’ to do instead).

The commitment to measure performance starts to wane, and isn’t maintained over the medium- to long-term.

Data collection is manual and time-consuming.

The performance data that is collected is inconsistent, incomplete, or inaccurate.

Performance data is not gathered in a timely enough manner to be used in making decisions.

People do not capture the data that the organisation needs them to capture.

Not enough resources are available to gather data for new performance measures.

Too much time is consumed in collating data and not enough time in analysing it.

It’s hard to get performance data out of data repositories such as database systems, spreadsheets and paper-based records.

Disparate data sources don’t communicate with each other, so it’s not possible to merge data.

Different groups within the organisation measure the same things in different ways.

There is inconsistency in how and when measures are reported.

Performance reports are time-consuming to produce.

Performance reports are overwhelming, cluttered, and hard to use and navigate.

Targets are set based on opinions.

Targets are set before the performance measures are even defined, or before the performance baseline is measured.

Conclusions about performance are based on comparing this month to last month, to the same month last year, or to a target.

Trend lines are used to assess changes in performance over time. (Yes, you read that right. Using trend lines in this way is a sign of a broken performance measurement process.)

It’s not clear who should take action based on the performance results.

There’s no time for discussion about why the performance measures are important, and what people can do to influence and improve the measures.

People fear that the measures will show poor performance.

Decision makers can’t agree on (or they argue about) the appropriate actions to take in response to performance data.

People cannot say how they would act differently if a measure’s result went up or went down.

Performance measures are ignored in decisions about which performance improvement actions or initiatives to invest in.

People cannot influence the performance results that their measures track.

Many of our colleagues and managers feel cynical about performance measurement because of their experience with these problems. Or they feel burned out from failed past experiences in trying to make performance measurement meaningful. So we often need to reinvigorate the support from our colleagues before we can get the training, systems or resources that we need to lead our organisations, companies or teams toward performance excellence. We need to give our colleagues new hope, a promise that there is an easier and more meaningful way to measure performance, and that doing so will truly transform the organisation.

The PuMP Mind Map exercise

The PuMP Mind Map is an exercise I have used many times to warm up executive and operational teams to the idea that PuMP is a fresh and exciting approach to meaningfully measure what matters. This is a quick and easy exercise to do, and people seem to love its cathartic effect. If we want to reignite hope in meaningful performance measurement, the PuMP Mind Map exercise is a great starting point.

Before I dive into describing how to do the Mind Map exercise, I’d like to offer a suggestion about when and how to use both this exercise and the PuMP activities described in the rest of the book. From here on, each chapter offers practical steps to take toward using PuMP in our organisations. Before undertaking any of these steps, including the Mind Map exercise below, I strongly recommend reading the entire book. We need to feel both confident that PuMP could be a good choice for renewing our organisation’s performance measurement process, and that we understand the basic principles of PuMP and its potential well enough to be able to speak with confidence and enthusiasm to others about the process. We will be most effective in helping to make the PuMP process work for our organisations if we approach it this way.

The first step to using the Mind Map exercise is to decide who we want to influence, and therefore who we should invite to do the exercise. The exercise takes between 30 minutes and one hour. It’s ideal to do the activity during a time slot set aside within an existing meeting, preferably one that has to do with planning, goal setting or performance monitoring so that the people who attend will be focused on the issue of performance measurement. We’ll need a whiteboard, or we can use a computer and data projector if we have mind-mapping software. We should make it clear to those attending that this is not a presentation, but rather an interactive activity.

Here’s the plan for the discussion that we’ll facilitate:

Step 1: Explain the purpose of meeting (5 mins) As we would start any meeting, we kick off the PuMP Mind Map exercise with a brief explanation of its purpose. For example, ‘I invited you to join me today to explore a completely different way to develop meaningful performance measures that solves a lot of the problems we’ve had in the past’. The purpose we express should be one that resonates with our colleagues, so we should choose our words deliberately.

Step 2: Discuss and collate problems with performance measurement (10 to 20 mins) On a flipchart sheet or a whiteboard, we draw the centre of our PuMP Mind Map, along with eight branches numbered 1 to 8.

These eight branches each relate to one of the eight steps in the PuMP Blueprint. But we don’t want to introduce PuMP in any way just yet, although we might want to have a ‘cheat sheet’ diagram in our notes with the eight branches labelled to remind ourselves. (We’ll discuss this in just a minute.) At the beginning, the discussion should be all about our colleagues’ experiences and feelings with performance measurement. They need to feel the pain before they can be open to a solution. And they need to see the patterns in their pain, so they can see the relevance of the solution.

We ask everyone to share the problems, pain and challenges with performance measurement that they are experiencing now, have experienced in the past, or are concerned could happen in the future. Some prompting questions might be:

What frustrates you about measuring performance?

What challenges are you experiencing with measuring performance?

Why do people avoid performance measurement activities?

Why aren’t we getting great value from our performance measures?

As each person shares a problem, write it down immediately on the Mind Map at the number (PuMP Blueprint Step) to which it most relates. This is where our cheat sheet comes in—we can refer to it if needed to jog our memory about what the 8 steps are and where we might place the issues that our colleagues list. We just do our best, and we don’t worry about perfect allocation of problems to PuMP steps; close enough is good enough. We can also put something on two branches if we can’t decide where it goes. We need to use the exact language that our colleagues use when they articulate the problems. We don’t try to summarise, rephrase, or make their comments more brief. We’ll lose the emotion and personality if we do, and we might even make our colleagues uneasy that we are altering their words to achieve a particular goal. If someone expresses a long complaint, we ask them to help us boil it down to a few words so we get it just right.

The illustration on page 49 is an example of how the map could look when it’s filled in.

Once the ideas start to slow down, we can acknowledge that some might be wondering what the numbers mean. At this point we summarise the basic struggle that links the problems listed on each branch, using terms that link to the eight PuMP steps. A suggestion of words we could use to describe those eight basic challenges is presented in the illustration on page 50.

Step 3: Debrief the discussion (10 to 20 mins) Now’s the time to introduce the PuMP Blueprint. We explain the eight steps, what they involve, and why they are important. Each step is designed to overcome the majority of struggles people have with performance measurement, which are the very struggles that our colleagues just mind-mapped.

As part of the debriefing, we explain that in addition to Step 4, the social side of performance measurement (e.g. politics, buy-in, learning) is managed through each of the remaining eight steps as well. PuMP has become well-known for the amount of buy-in it creates from even the most cynical members of an organisation.

We also explain how the PuMP Blueprint is a framework for a team-based approach to performance measurement, and that it’s supported with clear instructions and practical templates that are easy to follow and work every time. (This will be easier to do after we’ve read the book first and are convinced that it’s true.)

unknown
PuMP Mind Map - Numbers only.mmap - 8/05/2014 -

unknown
PuMP Mind Map EXAMPLE.mmap - 19/02/2014 -

Then we ask for feedback and comments. Our approach to this entire exercise at this point, including receiving feedback, should be a feeling of ‘holding the space’ for our colleagues to talk about measurement in an honest, open and different way. We do whatever we can to encourage that.

At the end, we can ask or suggest where would be a good place to pilot-test the PuMP approach. And then we ask what kind of results our colleagues would like to see to be convinced that PuM P is a successful way for the organisation to go. What kinds of struggles would they see vanish? What qualities would their new performance measures have? What impact would a new approach to measurement have on executing strategy, reaching goals and hitting targets? What would new measures mean for the mission and vision of the organisation?

The ideal outcome of the PuMP Mind Map exercise is a decision to pilot-test PuMP somewhere in the organisation, with support from management to monitor the pilot and evaluate its success after the eight steps are complete.

Take Action: Get support for your performance measurement overhaul

You can download an agenda and Mind Map template to conduct your own PuMP Mind Map exercise with your colleagues. Go to the book’s website (www.pumpblueprintbook.com) to get these templates.

Chapter Summary

The struggles people have with performance measurement are widely shared. Some are obvious to everyone, and others are not even on people’s radar as problems that sabotage performance improvement. If our performance measurement processes don’t produce meaningful performance measures that drive performance improvement, the reason is that the process is not designed well. Specific symptoms give clues as to where our performance measurement processes need improving or redesigning.

PuMP was designed to directly tackle the most common struggles people have with performance measurement, and provide a framework for improving or redesigning performance measurement processes to dramatically reduce those struggles.

We can start the process of getting support from our colleagues for trying PuMP by inviting them to mind-map the problems with the organisation’s current measurement process. We then describe how PuMP works, ask what kinds of changes they’d like to see in order to have confidence that PuMP would work for the organisation, and propose a PuMP pilot.

Are you ready to take a look at your performance measurement process with a fresh perspective and a practical attitude? Then let’s start planning your 8-week performance measurement process improvement.

3

The Eight-Week PuMP Blueprint Pilot Project

‘Sow a thought, reap an action; sow an action, reap a habit; sow a habit, reap a character; sow a character, reap a destiny.’

— Chinese Proverb

‘An ounce of action is worth a ton of theory.’

— Ralph Waldo Emerson

‘The way to get started is to quit talking and begin doing.’

— Walt Disney

The obstacles in the way of making performance measurement really work for us can be buried in any (or all) of the eight steps of the performance measurement process. Getting better at measuring performance means looking at the unique measurement process in each of our organisations as it is now, and searching for clues about what those obstacles are. The obstacles won’t be the same in every organisation; we all have our own individual strengths and opportunities for improvement. But in every organisation, most of the obstacles to excellent performance measurement will be found by examining the existing measurement process in light of the eight PuMP steps. So that’s where we’ll start to look for the obstacles and fix them.

The rest of this book is structured around the eight steps of the PuMP Blueprint. This chapter describes some preliminary steps and an overview of an eight-week pilot project to try out PuMP in an organisation, as well as some frequently asked questions. The next eight chapters lay out in detail the eight steps of the PuMP Blueprint for implementing good performance measures quickly and easily.

PuMP is not the only way to go about performance measurement. It is the way that reflects my journey and my experience, and it’s helped thousands of others who didn’t have a way or a way that was working for them. So we need to put our preconceptions and experiences to one side, and try this way for ourselves. We’ll start with a little preview of each of the Performance Measure Blueprint activities to survey the big picture before we dive into the details.

Why eight weeks?

It’s quite easy to underestimate just how much it takes to transform an organisation’s approach to performance measurement. Organisations have many established bad habits that cause the very problems with performance measurement that we would love to solve. A few of those bad habits relate to not giving it enough time, and not being deliberate enough in the way we go about choosing, implementing and using measures to improve performance. If we start yet another performance measurement program without addressing these bad habits, and giving them some time to change, we doom ourselves to undertaking yet another one of those performance measurement implementations that failed. So the alternative I recommend is to start with a pilot, or a quick and focused PuMP implementation, that gives the participants a chance to learn new thinking, unlearn bad habits, and fine-tune the approach itself before going large later on. That’s what the eight-week PuMP Blueprint Pilot Project is all about. It’s a way of opening the windows, letting fresh air in, and helping our organisations start fresh instead of repeating the same bad habits.

Our emphasis through the eight-week PuMP Blueprint Pilot Project (let’s call it the PuMP Pilot) is learning through action—not aiming for perfection the first time through (or ever, to be perfectly honest). It’s helpful to think about ‘continuous improvement’ for our performance measurement processes. Each time we apply the process we reflect, review and make little improvements. This means that our first, second or even third PuMP implementations will involve two levels of learning: learning the PuMP techniques themselves, and how to facilitate the PuMP techniques in a way that works for our individual organisations.

Even as we explore these two levels of learning, it’s important to maintain momentum through an implementation for a few reasons. First, if we keep the pace up we keep the energy high, and people will have a better experience than if we take months to get our first performance improvement. Second, if we fail fast we can learn fast. This means testing how PuMP works with our individual organisation, and discovering ways to fine-tune how we apply to suit the organisation’s individual needs so we will have more success on a bigger scale later. Third, by moving quickly we can arrive at a quick win much sooner. It’s the quick wins that sell the benefits of performance measurement to those who are sitting on the fence.

Chris, from a health and sciences university in the United States, is one of the people who learned the PuMP Blueprint to an advanced level. Chris’ reflection on his first few PuMP implementations is helpful to read as we’re starting out:

When I first started out on this journey, I endeavored to take eight teams through this process in the course of 12 weeks. It seemed like a tenable goal. I laid out an aggressive schedule that had each team meeting for one hour every two weeks. It seemed like it would be possible to take the teams through the process little by little without too much diversion from their normal duties. How wrong I was.

Looking back on it, I now realize it was too little stretched out over too long a period of time, and the timing wasn’t quite right. The teams were just heading into the busiest season, end of fiscal year, which meant that there was little time left over for improvement initiatives and there were many cancelled meetings as a result. Also, the manager was unable at that time to participate in the meetings, which left a disconnect of understanding about the process, as well as a general lack of commitment from the teams due to no visible management participation. In addition to that, it was just too much for me to attempt to carry on my normal duties and lead this initiative by myself.

Going forward, I will schedule bigger blocks of time (maybe four hours a session), and hold the workshops once a week rather than every two weeks with a completion date no later than six weeks for the training and the initial suite of measures. I will also try to get earlier “wins” on the board to fuel the momentum and buy-in from the team. And most importantly, I will insist on a commitment from the manager to participate in the sessions and the discussions prior to commencing the engagement. Without that, it’s hard for a team to recognize the initiative as a priority when there are so many other things competing for their attention.

There is one final and very practical reason why I’m recommending that we take eight weeks for our first few PuMP implementations. There are eight essential steps in the Blueprint, and devoting a week to each step in turn is a nice way to pace our progress. And we’ll make the best progress if we start small and focused.

Start small and focused

Where we start with our performance measurement projects has a great bearing on their success. We don’t want to add to people’s cynicism about performance measurement by putting a lot of effort into measuring something that doesn’t matter and doesn’t make an important and valued difference. If there’s still some uncertainty or cynicism about performance measurement in your organisation, don’t be in a hurry to implement a corporate-wide performance measurement approach in the early months. You’ll get more traction by starting your performance measurement journey with a few PuMP Pilots.

What is a PuMP Pilot?

A pilot is a small, focused trial run to test an approach before deciding if and how to implement it more widely. Pilots are often used to test new business procedures, technologies, medications and even market research questionnaires and surveys. A PuMP Pilot gives us the opportunity to truly learn the PuMP techniques. Some of these techniques are going to challenge misinformed beliefs and bad habits that both we and our colleagues have, and it takes time for beliefs and habits to change. Very often people have a history of negative experiences with performance measurement. A pilot gives them an opportunity to reframe what measurement is really about before they are asked to fully commit to it. Performance measurement has a profound impact—whether it’s positive or negative—and it ought to be treated as a change process at least as much as a technical one.

With a pilot test of the PuMP Blueprint, we’re not only implementing the approach; we’re also evaluating it. So even though the pilot is a quick and simple version of what we hope to implement more widely later, we should not be blasé and think that our PuMP Pilot doesn’t need careful planning. It does if we’re going to get the most leverage from it. As we create the project plan for our PuMP Pilot, here are some suggestions:

Start small. Choose just one team or one process or one strategic goal to develop measures for. Don’t give in to the temptation of scope creep. The more ruthlessly we focus the first time through, the more success we’ll have and the sooner we’ll have that success.

Involve a small group of about four to seven willing volunteers. Working with volunteers means we don’t have cynicism fighting the momentum the first time through. The cynics will be more valuable in subsequent performance measure projects. Time and again I’ve witnessed them becoming the strongest advocates for PuMP and performance measurement because of its inclusive and logical approach, and the great results from the measures. But for now we want to create all the conditions possible to support our short-term success by working with those who are willing and interested.

Set a short timeframe and stick to it. I find that with a small focus group, an eight-week time frame is realistic to select, implement and start using the first meaningful performance measures.

Have a plan that draws on the simplest application of PuMP, but be sure to work all the way through the process. Don’t stop after designing some measures. Bring them to life and use them. Test their true value, and give people the satisfyingly complete experience of what performance measurement is really all about.

Build in time at the end to reflect, review, and design the way onward and outward with the next phase of PuMP implementation.

Write up the pilot experience as a case study, with the spotlight on the performance improvements achieved and not just on the measures themselves. Share the case study with the people who would ideally be involved in the next phase of PuMP implementation. Encourage the interest and ownership to grow organically—don’t force it.

We might need to do a few pilot runs before a critical mass of interest and engagement builds and we get enough leadership support for corporate adoption of PuMP as the organisation’s performance measurement framework. And if we’re well organised, we can carry out a few PuMP Pilots within a matter of months. If we’re very focused on just one small implementation of PuMP first, we’ll find eight weeks is a well-paced, momentum-maintaining timeframe.

Choose the focus for the first PuMP Pilot

We don’t want to develop performance measures because we think we have to or should. We measure performance only when there is a performance result that we need to manage. And by manage, I mean ‘maintain’ or ‘improve’. I’m fond of saying that we should measure only those performance results that we should, can and will do something about. If we’re not going to take action on any signal a measure will give us, then we shouldn’t bother to measure it. Or if we do measure it, we shouldn’t call it a performance measure. Its role might instead be to provide context or cause analysis for a performance measure, or simply to be a piece of information that someone needs for a purpose other than managing performance. For example, there are many measures in a Profit and Loss statement that we need for tax purposes, but that doesn’t make them all performance measures.

So when we decide to start measuring performance, or to start with PuMP, we need to make sure we focus on results that we can, should and will do something about. And where we find those results depends on whether our organisations do or don’t have a defined strategic direction.

WHEN THE ORGANISATION HAS A DEFINED STRATEGIC DIRECTION

Many organisations have a defined strategic direction: a corporate plan, which is a collection of corporate goals or objectives that need to cascade down throughout the organisation or perhaps already have been disseminated. If the organisation has a defined strategic direction, that strategy is the place to go looking for the focus of our PuMP Pilot.

For example, if we want to start performance measurement with the executive team, then the focus for our PuMP Pilot would be helping the executive team find meaningful performance measures for the performance results implied by the current strategic direction. If the organisation’s strategy has six goals, that implies that it is aiming for a set of six or more performance results. We want to develop the performance measures for those goals.

Another example is a team or department that has been made responsible for implementing part of the corporate strategy, such as the service delivery team, the information technology team or the maintenance or operations team. Whichever team or department it might be, the focus of our PuMP Pilot could be helping that team come up with the first five or six performance measures that align their goals with the corporate strategy.

WHEN THE ORGANISATION DOES NOT HAVE A DEFINED STRATEGIC DIRECTION

Some organisations have not defined a strategic direction, or are still developing one. If we don’t want to wait for the corporate strategy to be ready before we start measuring performance, we can focus our performance measurement project on results that are ‘mission-critical’ or intrinsically important to the organisation’s very purpose. They might be obvious, like delivering parcels on time is to a courier company, or patient health is to a hospital. Or they might not be obvious, in which case we can find them out simply by asking our customers.

Here are three tips for how to choose the focus for a PuMP Pilot.

The first tip is to choose one known performance problem. For example, we might already know that there are lots of complaints from customers about not being kept informed or updated about their product orders or the progress of their enquiries. We can target that problem as the focus for our PuMP Pilot. We start by writing a goal statement that describes how things should be, and we use the PuMP Pilot to measure progress toward achieving that goal.

The second tip is to ask our customers. These could be internal or external customers depending on whether we are working with a support division or a customer/external-facing division in the organisation. We can ask a random sample of customers a question like, ‘If we were to get better at three things, what three things would you most want us to get better at’? These three things would be drivers of customer satisfaction, and we could pick the one thing mentioned most often as the focus of our PuMP Pilot.

The third tip is that productivity is always a good thing to work on. I’ve heard it said that if we were to focus solely on reducing the cycle time of what we do, almost all other performance results can improve as a result. This makes some sense, because if we reduce the cycle time of our business processes, we have to strip out re-work, reduce waste, and make sure that what we are doing are the essential things. So cycle time could be a place to start as the focus of our PuMP Pilot.

Form the Measures Team

The Measures Team is a temporary grouping of people who represent the business unit or the business process with whom we want to implement PuMP. This Measures Team acts on behalf of everyone they work with until it’s time to create their Measures Gallery, at which point everyone else can get involved.

How we form our Measures Teams will have a very big bearing on the success of our PuMP Pilot and subsequent PuMP implementations, so it’s important to be deliberate in carrying out this step.

The optimal size is between four and six people

Measures Teams work most effectively—with the best rapport, focus and speed—when we limit their size to between four and six people. The only exception to this rule is when we are working with an executive team or senior leadership group of an organisation or business to develop performance measures for the organisation’s corporate strategy. In this case, the ideal is to include every executive in the Measures Team. It’s important that they all share the same understanding of the strategic performance results, and all share ownership of the measures that they design. Including the whole executive team also shows the rest of the organisation how important it is to measure what truly matters.

Why no more than six people? Experience keeps confirming that when we have more than six people on a Measures Team, the process dramatically slows down. I’ve personally facilitated Measures Teams as large as 12 people, and the results just aren’t as good. The number six is almost like a tipping point. Any more people than that and the dialogue gets too complex, too many people have too many things to say, and it starts taking seven or eight hours to get the first draft of the Results Map done. Spending that kind of time dramatically slows the momentum, and burns up the limited amount of energy people have for their first experiences with better performance measurement. It’s really important to instead keep the energy high and get results quickly so that the success gets consolidated, which creates excitement and energy to keep it going.

Why no fewer than four people? If we have fewer than four people on a Measures Team, we don’t get enough dialogue and debate, enough banter, and enough testing and checking with each other. There aren’t enough fresh perspectives and ideas coming into the conversation, and the resulting measures are just mild improvements on the status quo.

Your role is to be the leader and facilitator

So who are we then? Are we part of the Measures Team, or are we something a bit different? I’m going to suggest that because we know something about PuMP, we are the leaders and facilitators of the Measures Teams. Our job is to hold the space for our Measures Teams to go through each step of PuMP in a series of workshops or meetings. What I mean by ‘holding the space’ is that we are giving them everything they need to easily apply the PuMP techniques in a way that’s really meaningful for them. We are not doing it for them, and we’re not telling them what to do. Instead we’re giving them information, showing them examples, guiding them, and teaching and leading them to apply the techniques of performance measurement. That’s really the role that we need to play. And if we are not the ones to take that role for some reason, then we’ve got to have someone who can, and who understands PuMP very well.

Another option is to consider including a group facilitator who has the skills to manage the group dynamics in the Measures Team while we focus on leading the team through applying the PuMP techniques.

Have a good mix of members

Once we’ve chosen the focus of our PuMP Pilot, then we should already have our first Measures Team member—the manager of the area for which we are planning to develop performance measures. In recruiting the remaining members of the Measures Team, there are a few considerations. And the first one is to start with volunteers—people who want to be part of the process.

If the planets line up, we will find a balance of people for our Measures Team who collectively have knowledge of things such as:

what data might be available or collected or ready that is relevant to our focus area

how the work processes function in the area that we are trying to measure

what the customers of that particular area or process need

whether there are any existing measures that this area of the organisation might already be using.

When we have a group of people who together have this collective knowledge, we can achieve a lot more during the PuMP process because we don’t have to constantly ask questions of other people who are not on the team.

Invite people to join; don’t mandate it

If being involved in performance measurement is mandated or insisted upon, people come to the process with the wrong energy. Think about it: when you’re told to be somewhere that you aren’t 100% convinced you want to be, you rarely turn up with an open mind, open heart and open arms, do you? Most of us turn up quite reserved, or even a little bit resentful. That’s not the energy we want to bring in to our PuMP Pilot, or any PuMP implementation we do. So we invite people to be part of it, but we don’t make them be part of it.

The manager must actively participate

The manager of the area for which we are developing measures can’t be allowed to get away with saying, ‘Yeah that sounds great. Off you go, Jeff. You do this for us’. The manager also has to give their time to the PuMP Pilot. What that means is that they personally attend the PuMP Pilot meetings and workshops, and also take on the role of sponsor, visibly and verbally supporting the PuMP Pilot. We cannot get great results without leadership involvement from the start.

In one of my clients’ PuMP implementations, where we were creating the performance measures for the new strategic plan, the CEO was unable to attend the PuMP Step 2 meeting about Mapping Measurable Results at the last minute, and the executive team continued without him. He then attended the Step 3 meeting on Designing Meaningful Measures, and ended up derailing the team’s hard work by revisiting and rehashing the Results Map. Not only was this very disruptive to the executive team’s focus on choosing performance measures, but it was disrespectful to the shared learning, dialogue and decisions they’d worked through to articulate their strategy in measurable terms. The CEO missed the true purpose of the Results Map because he hadn’t shared the same experience as his executive team. And his changes to the Results Map unravelled all the great work they’d done. So although I just said that we should not mandate that anyone participate in the PuMP Pilot, we have to make sure the manager of the area we want to measure understands the importance of his or her involvement right from the start.

It’s about measuring processes, not measuring people

A final point about forming the Measures Team, which might go a long way to helping us get potential team members interested and willing to accept the invitation to join: we need to make it clear that the process is not measuring them. It’s not about measuring their performance. It’s about measuring the performance of the process or the business unit that they work in. This helps to take the threat out of performance measurement, particularly if people in the organisation have been stung before by performance measures being used to ‘manage’ them. Measuring people is fraught with risk, especially in organisations that don’t even know how to measure process performance. So we establish the organisation’s measurement skills by focusing on process performance before we worry about whether or not we should be measuring people’s performance.

Schedule your PuMP Pilot

As I mentioned earlier, before we start working with PuMP in our organisations it’s important to first read through this entire book so we have the big picture of how the PuMP Performance Measure Blueprint works. We can go into this program without being experts, but we should not go into it without being informed.

Once we understand the PuMP Blueprint, we know why we need to schedule between 90 minutes and three hours for each step.

Here’s the series of meetings that we schedule for a PuMP Pilot:

Week 1: Get prepared, and warm up the team with the PuMP Diagnostic

Week 2: Make the organisation’s strategy measurable

Week 3: Design the measures (but don’t brainstorm them)

Week 4: Engage stakeholders, and build buy-in

Week 5: Define the ‘bring to life’ details of the measures

Week 6: Design useful and usable performance reports and dashboards

Week 7: Respond to what the measures are really telling us

Week 8: Use performance measures to improve performance

Let’s take a closer look at what we’ll be doing in each of these weeks.

Week 1: Get prepared, and warm up the team with the PuMP Diagnostic

The theme of Week 1 is to warm up the Measures Team so they feel excited, curious and confident about the journey to which they are committing. This is Step 1 of the PuMP Blueprint: Understanding Measurement’s Purpose.

When we are taking a new team through a PuMP implementation, it’s a little like leading people to do something when they don’t yet appreciate its importance. Warming them up is the first stage of helping them appreciate the value of performance measurement done well, what good performance measurement involves, and how some of their past experiences and practices need to be changed. Chapter 4 helps us lead the Measures Team to the first stage of this appreciation by using the PuMP Diagnostic: a tool that guides a discussion about what good and bad performance measurement practices are, and helps the team define the baseline measurement approach as it is before any changes are made.

Week 2: Make the organisation’s strategy measurable

The theme of Week 2 is to decide what specific results are worth measuring. This is Step 2 of the PuMP Blueprint: Mapping Measurable Results.

This week is not about what the measures will be, so the team will need to exercise some restraint until Week 3 where the measures themselves will be the focus. Usually the things most worth measuring in an organisation are decided through strategic and operational planning processes and end up being written as critical success factors, objectives, goals or priorities. But the way we typically write these goals or objectives is a huge hindrance to measuring them meaningfully (or at all). To define goals that are measurable, we need to make a fundamental change in the kind of language that is usually used for writing strategy. That’s why Week 2 is not the time to talk about measures. If anyone starts brainstorming measures, getting hung up on what the measures should be, or asking the question, ‘Yeah, but how do we measure that?’ we must insist that question be saved for Week 3, because now is not the time to talk about that. Chapter 5 will explain more about why it’s so important to keep the discussion about what’s worth measuring separate from the discussion about what the measures will be. We’ll learn the PuMP Measurability Tests and the PuMP Results Map techniques for making things measurable in a way that’s easy to communicate and easy to cascade throughout your organisation too.

Week 3: Design the measures (but don’t brainstorm them)

The theme of Week 3 is to design measures that give the best evidence of the results that the group decided were most worth measuring in Week 2. This is Step 3 of the PuMP Blueprint: Designing Meaningful Measures.

This week, the group avoids using approaches that have likely been used in the past: brainstorming, figuring out what can be done with the data the organisation already has, measuring the same things that have always been measured, benchmarking to find what others measure, and hiring consultants to tell the organisation what to measure. These approaches don’t help us think through the kind of evidence that will let us know the degree to which our results have been achieved. There’s a better way, and in Chapter 6 we’ll learn all about it, including a practical five-step PuMP Measure Design template.

Week 4: Engage stakeholders, and build buy-in

The theme of Week 4 is to hold the space for the broader circle of stakeholders to engage with the measures. This is Step 4 of the PuMP Blueprint: Building Buy-in to Measures.

Not everyone gets to be actively involved in the selection and development of performance measures. But everyone needs to feel at least some level of ownership of the measures that will impact them, or the measures on which they will have an impact. The greater the ownership and buy-in that people feel toward the measures that relate to them, the greater the value those measures will bring to the organisation. Chapter 7 will explore in detail the fundamentals of engaging people in performance measurement that are the theme of Week 4, and we’ll learn a great method called a PuMP Measure Gallery, which makes it easy, fun and quick for people to contribute to the development of the measures that truly matter.

Week 5: Define the ‘bring to life’ details of the measures

The theme of Week 5 is to define the operational details of the performance measures that our groups have chosen and make it clear how to implement the measures (or ‘bring them to life’), as I like to say. This is Step 5 of the PuMP Blueprint: Implementing Measures.

Before we implement each measure, we need to determine how it is calculated, identify the data we need, and define several other important aspects that will constitute an action plan to ensure that the right thing is measured in the right way. This means keeping something like a performance measure dictionary for our organisations. We’ll learn all about this PuMP Measure Definition template in Chapter 8.

Week 6: Design useful and usable performance reports and dashboards

The theme of Week 6 is designing graphs and reports or dashboards that facilitate valid interpretation of the measures and their use in decision making. This is Step 6 of the PuMP Blueprint: Reporting Performance Measures.

Spreadsheet and dashboard applications don’t always know the best ways to visually present our particular performance measures. The default charts in these programs often don’t answer the three most important questions that a performance report should answer—‘What is performance doing relative to target?’, ‘Why is it doing that?’ and ‘What do we need to do to move it closer to target?’ Performance reports need charts that provide enough context for the measures, highlight real signals of change in the measures, and throw some light on what actions are most appropriate to improve performance. Chapter 9 will give us the guidelines to create useful and usable performance reports and dashboards using the PuMP Performance Report template, which will do the job quickly, easily, and gratifyingly.

Week 7: Respond to what the measures are really telling us

The theme of Week 7 is to start looking at our performance data in a new way to ensure that we see the real signals of change and stop reacting to noise. This is Step 7 of the PuMP Blueprint: Interpreting Signals from Measures.

When we know how to really interpret what our performance measure signals are, we eliminate two very large problems in decision making. First, we stop wasting time chasing problems that aren’t really there. Second, we don’t waste time and money allowing problems that are real to get worse simply because we weren’t aware of them. In Chapter 10 we’ll discover the simplest and most powerful method of highlighting real signals in our measures: the XmR chart. I’ll demonstrate how this chart can transform the insights we get from our measures using the step-by-step PuMP Interpreting Signals from Measures technique.

Week 8: Use performance measures to improve performance

The theme of Week 8 is to use our performance measures to guide analysis and find high-leverage actions to reach our targets. This is Step 8 of the PuMP Blueprint: Reaching Performance Targets.

We make the best use of scarce time and resources when we pinpoint the root causes of performance results (i.e. getting deeper than the symptoms) before we decide what actions to take. In Chapter 11, we’ll set performance targets that encourage lasting improvement gains, and choose improvements that have a large and measurable return on investment. This chapter will take us through the PuMP Reaching Performance Targets technique to instil the habit of continuous performance improvement.

Some questions frequently asked at the outset

There’s never only one right way to get things done, and in the spirit of preparing us for whatever way we each choose to proceed with the PuMP Blueprint, let’s look at some of the most asked questions from real people as they embarked on their first PuMP implementation.

Is eight weeks really enough to do this?

Yes, eight weeks is enough to test PuMP, engage people, and demonstrate the return on investment of a good approach to performance measurement. The purpose of the eight-week pilot is not to perfect the implementation of PuMP, but to learn the principles and techniques and apply them the best we can. We won’t succeed if we wait until we have the time to perfect each step. And it’s easier to course-correct when we’re moving than when we’re standing still. Action learning is our motto for these first eight weeks.

It is hard to overstate the importance of getting an obvious and valuable performance improvement as quickly as possible. A little success does a world of good by building buy-in, fuelling enthusiasm to continue, and demonstrating a return on investment. Performance measurement efforts that drag on too long often lead to little performance improvement and a whole lot of cynicism and waste.

Should I start at the top?

Absolutely start at the top if you can. If we have the CEO or senior executives in favour of supporting better performance measurement and performance improvement, and development of a strong performance culture, then we should definitely start the process by working with them to devise great strategic (corporate-level) measures first.

If the CEO or senior executives aren’t sold on the idea yet, we do what we can to encourage them and garner their support. We show them examples of good performance measures that led to significant performance improvement, the waste hidden in the existing performance measurement processes, and how we can start turning performance around in as little as eight weeks. In an international not-for-profit organisation, a typical retort from managers was that they didn’t need performance measures because they could tell what was going on by walking around. To help sell these managers on the idea that performance measures could indeed help them, performance analyst John would investigate performance measures that were showing adverse trends and start a discussion about the causes. One such performance measure was Contract Preparation Administrative Time, and it had been steadily increasing over the years. John discovered that the increase was because the line manager had been setting each year’s target based on the actual performance of the previous year. The targets had increased over five years from 40 days to 72 days. When he discussed this with the Director responsible for the measure, the Director thought the increase was due to higher levels of workload. They turned around the target setting problem and improved Contract Preparation Administrative Time back to 48 days, with a view to continue reducing it further.

A good default for where to start your performance measurement campaign is to start where the true energy is and grow outward, upward and downward from there. If a middle-level manager, strategy manager or operational team is desperate to find some meaningful performance measures to help them solve a problem, then we start the performance measurement process with them. Their success (if we aren’t too shy to share it with anyone who’ll listen) will spread like a ripple effect and help stimulate interest in other parts of the organisation or business.

Do I need an expert facilitator?

You definitely need a facilitator, and this book will give you enough support to do a reasonable job of being that facilitator yourself. Many people are already doing it, so rest assured you won’t be the pioneer.

If you want to become an expert facilitator of the PuMP Blueprint, a ‘PuMP Professional’ or even a ‘PuMP Consultant’, there are some more skills to learn than are presented in this book. You need to practice, make mistakes, learn from those mistakes, and get a few good implementations of the Blueprint under your belt. That takes another level of training.

If you haven’t got time, or you want to fast-track your journey to be your organisation’s PuMP Blueprint expert facilitator, you can work with a licensed PuMP Consultant to speed things up. (You can find one at www.staceybarr.com.)

Can I create the measures by myself?

You can, but please don’t. One of the worst things you can do is design and build your organisation’s or business’s performance measurement system alone, present it to everyone else and invite them to take ownership. They won’t, because it’s not natural to feel ownership of something you haven’t had a hand in creating. Approaching the process this way will exhaust you, add to the cynicism people already feel about measurement, and waste everyone’s time. (If those things don’t happen immediately, it won’t take long before they do.)

A very large part of building buy-in to performance measurement (particularly given its stigma as challenging, boring and threatening) is to involve people from the very beginning in selecting, creating and using performance measures. Please don’t try to do it alone. Find a handful of colleagues, form a Measures Team to do the bulk of the work, and periodically open up for feedback from everyone else.

How many PuMP Projects is it possible to focus on simultaneously?

I know of many people who stick with just one PuMP Pilot to start with until they have confidence and skill to handle more. The most I’ve seen someone handle was eight teams in parallel, but that person was running PuMP Pilots full-time and doing one PuMP step per month rather than one per week.

When we are going at fast-track pace, and it’s our first time through the PuMP Blueprint, it’s best to limit ourselves to working with just one or two teams. But after we’ve completed one or two successful fast-track PuMP Pilots, we can get a little bolder and take on more.

If we are working full-time at performance measurement, or want to move slower than fast-track pace (e.g. fortnightly instead of weekly), we can try three to four teams concurrently with each team doing one PuMP step per week.

If we can work full-time on performance measurement, and facilitate three to four teams doing one PuMP step per week, then we could also set up a fortnightly schedule where three to four teams begin in week 1, do their second PuMP step a fortnight later in week 3, and so on. We could then start a second group of three to four teams in week 2, who would also complete their PuMP steps each fortnight. With this staggered fortnightly schedule for the two groups of teams, we could take six to eight total teams through PuMP, facilitating alternating groups each week.

If we are doing performance measurement part-time, we should limit ourselves to one or two teams at first. Then, after we’ve done our first PuMP Pilot, we can better judge for ourselves the amount of time and number of teams that we can handle.

Will it work in my industry/sector/culture/small business/large corporation?

Absolutely. Performance measurement, and the PuMP Blueprint, will work in your industry, in your sector, and with your culture. Business planning works in all industries, sectors and cultures. So do human resource management, financial management, safety management, recruitment and selection of employees, and hundreds if not thousands of other business processes. There isn’t a different performance measurement process for different industries, sectors or cultures. Each organisation will have different strategic goals because that organisation is unique within its industry or sector. But the business planning process that brought the organisation to those goals could be identical to one used by many other organisations. So yes, each of us will measure different things because each of our organisations is unique. But the fundamental approach to selecting, creating and using measures is the same.

We can use the PuMP Blueprint in any industry, sector or culture to help us quickly and easily find the measures that are meaningful to our unique organisation or business. PuMP doesn’t tell us what to measure. It helps us determine how to measure it.

How do I know if my organisation is ready?

We can look for a few signs that an organisation is ready to start taking performance measurement seriously.

One sign is a performance problem that qualifies as a ‘burning platform’, such as the threat of being outsourced or downsized, losing major customers to competitors, or instigating big budget cuts. A performance problem that needs fixing requires good information in the form of performance measures to define, diagnose and monitor performance in support of solving the problem.

Another sign is a champion either in a senior role or with some influence who will make the space and time for a team to work on performance measurement. A CEO or senior executive who actively supports better performance measures is the ideal starting point. But any well-respected leader who has bought into the notion that meaningful performance measures are worth the trouble can open the doors.

A third sign—or more an indication that your organisation has no choice—is an external body such as an industry regulator, Board of Directors, government agency or similar insisting that the organisation sort itself out and start measuring meaningful things. If we handle this situation well, it becomes an opportunity to give people a surprisingly meaningful experience in measuring performance while dealing with those external demands.

Do I need management or executive support?

We’ve already alluded to this a few times, so we know the answer. If we can get management or executive support for our performance measurement initiative, then we have the best chance of it succeeding and spreading throughout the organisation.

But if we can’t get senior leadership support straight away, then we need to work with those who have the energy and the need for better performance measures, and then use those successes to help senior leaders see the real value of measuring what matters. Leaders tend to give more value to real business impact than to eloquent theories, so rather than tell them how it works we show them.

If our leaders are staunchly opposed to us spending any time at all on performance measurement (or a more meaningful approach to it) then we may need to ask ourselves if we’re working in the right organisation or business.

What if my organisation doesn’t have a strategy, goals or objectives?

Why do we want to have performance measures when we don’t have a strategy? What do we want to monitor with measures other than the things that matter most for organisational success?

Performance measures are at their best when we use them for performance improvement, not for rationalising the status quo, defending the case for more resources or covering our bums around bum-kicking managers. Sure, we can use performance measures for these purposes (and many do). But we won’t get the best return on our efforts to measure performance unless we focus the use of those measures on improving business processes so they produce better results for the organisation’s stakeholders. We won’t get the real value of performance measures unless we use them to help us continuously improve the capability of our organisations to excel at what they do.

A strategy usually consists of a collection of goals or objectives stating the results that the business or organisation wants to attain most in the short- to medium-term (i.e. one to five years into the future). Creating great performance measures starts with measuring the strategic goals, so if the organisation doesn’t have a strategy or planning process we’d be wise to find one and use it. Many organisations have found the Balanced Scorecard a useful method for designing a compelling strategy. (I particularly like the nine-step method developed by the Balanced Scorecard Institute.) More generically, the choice of performance measures should be preceded by a process that includes development of a vision, a mission, values, a business environment analysis, objectives and strategies to achieve the objectives.

What are the signs that I should quit before I start?

If we’re certain that people will use the measures as tools to judge other people, we should not bother starting. We’ll just make a bad situation worse. When people feel that measures are solely used as rods to flog their backs, then there’s a culture of fear and defensiveness. Performance improvement works only in a climate of trust, openness, experimentation and learning—the complete opposite of fear and defensiveness. In a culture where measures are likely to be used to judge people, we need to work on the culture first. We need to let people know it’s okay to make mistakes if we learn from them, and encourage teamwork and problem-solving that delves into root causes rather than blame.

Similarly, if people just don’t understand the concept of performance improvement, and working on the business instead of just in it, that’s another sign that it’s too soon for performance measures. Some organisations or businesses have a culture in which people turn up to work each day, do the hours, and get paid. It’s almost as if the employees feel they are selling their hours to the organisation. They don’t feel any responsibility for improving the organisation’s success or results. Until we can turn that culture around into one that understands that performance is measured by results and impact and not by effort, a performance measurement campaign will be resisted or treated with cynicism.

Another sign that we should quit before we start a performance measurement campaign is when people insist they cannot make time for it. They might say, ‘We’ll just have to fit it in somehow.’ But that’s not good enough. If that is the prevailing attitude, we’ll just be creating another failed attempt to produce great measures because then people won’t turn up at meetings, meetings will be rescheduled, people will cut corners to shorten the process, energy will be low, resentment will hang in the air and momentum will slow to a crawl. We need to be confident that there is time for performance measurement, even if it means that something less important is delayed or cancelled. And there will always be something less important than measuring performance. (Why? Because if you’re not measuring performance, you’re not managing it. And if you’re not managing it, there will be things you’re doing that equate to waste or rework. So you can stop doing these things and measure performance instead.)

What are the absolute minimum conditions for success?

A lot of factors will make our performance measurement programs easier and more successful. But I believe, and have certainly experienced, that we can manage a lot of these factors along the way. Building our confidence and proficiency in facilitating PuMP, involving the right people, choosing the simplest technology to implement our measures, finding the best measures, aligning measures with strategy and so on will improve with multiple iterations through the PuMP Blueprint. But I’ve found three conditions that have the greatest bearing on the success of introducing a new approach to performance measurement, and we want to have these conditions met before we start.

1. We have at least one goal that matters enough to measure. We should not start a measurement program without any kind of goal, objective, priority or result that we want to achieve. Performance measures are evidence of performance results, and performance results are usually defined in business plans. On the flip side we should not start a measurement program with too many goals or objectives to measure because we’ll quickly lose momentum. When we and our colleagues are learning a new process, some of our energy and attention will be consumed by adjusting to that new process and unlearning bad habits that have caused the performance measurement problems we’re trying now to fix. So we choose just one goal and start with that. We can tackle more later on, after our first eight-week PuMP Pilot.

2. We have a leader to give his or her support to our PuMP implementation. This leader might be a manager or executive, or someone with a great deal of influence with managers and executives. And by support, I mean that this person will turn up at each PuMP meeting and contribute actively in a very egalitarian way while respecting everyone else’s contributions at those meetings. Without the support of a leader, it’s likely that people will let other urgencies take priority over turning up to our PuMP meetings. They’ll also be less confident in making decisions about what to measure for fear those decisions will be overturned by their manager. At the same time, if the leader’s contribution at the PuMP meetings is too overpowering or domineering, we won’t get the buy-in from everyone, or the best ideas for what is most worth measuring. We need to find a leader who has a passion or desire to achieve a goal that better measurement will help achieve.

3. The people who will be involved in the PuMP implementation have committed the time to do it. The amount of time can vary, but we can achieve a very focused eight-week PuMP Pilot with as little as two hours per week. People are too busy at work, and have been for the past decade. We have too many goals and projects, urgency equates to priority, stress levels are steadily rising, and most of us work many more hours than we’re paid to work. So it could be that we will need to find something that people can stop doing to make real time available for measuring performance instead.

It’s worth noting that without good measures of performance, people will never know what the best use of their time is. So they will always feel like they’re working harder and harder, and settling for less and less. When we take this broader perspective, we can see why there has to be something less important and less urgent than spending the time to develop just a few good performance measures. So when our weekly PuMP meetings are scheduled into people’s diaries, and they give us signs that they’re looking forward to those meetings and aren’t feeling pressured by other urgencies, we’re all set.

Take Action: Plan your PuMP Pilot

Download the PuMP Pilot Project Plan template from the book’s website (www.pumpblueprintbook.com), and use it to plan your first eight-week PuMP implementation. Collaborate with your Measures Team to organise a two-hour meeting each week, and then you’ll be ready to get the ball rolling in Chapter 4.

Chapter Summary

We can get very proficient at performance measurement by starting small and moving fast. The eight-week program for our first few PuMP Blueprint implementations gives a structure to keep us moving quickly and a purpose to help our first few successes blossom into much more.

We should frame the first few eight-week PuMP implementations as pilots, and deliberately reflect and review how the approach works for us so we can improve our approach to measurement, not just the measures themselves. This also gives some space to allow people to change their beliefs and attitudes to measurement.

A few important conditions for success need to be met, even in the first few implementations are. Focus on something truly worth measuring, and don’t let attention broaden or wander away from that focus. We need a manager or leader who will support us. The best support a manager can give is their physical presence, and active and egalitarian participation in each step of the PuMP Pilot. And we need to make time to do it, allocating two hours per week for the eight-week program.

Now it’s time to start working on our performance measurement processes with a simple discussion to help the Measures Team truly understand the journey we are embarking on together.

4

Step 1: Understanding Measurement’s Purpose

‘Change might not be fast and it isn’t always easy. But with time and effort, almost any habit can be reshaped’.

—Charles Duhigg, The Power of Habit

When we start a PuMP Pilot for the first time with a group of people, we need to remember that they are likely still holding on to outdated beliefs and bad habits to do with performance measurement. They are not starting from the same understanding that we are. They need warming up before we launch them into following new techniques and filling out templates in pursuit of new and improved performance measures.

So Step 1 in the PuMP Blueprint is about warming up. An athlete doesn’t arrive at the starting blocks for their event without warming up first. Rather than rushing our colleagues into the process, we help them warm up. And the ideal kind of warm-up is something that will get them curious about why PuMP is our chosen approach, and set up the expectation that performance measurement is going to be very different this time.

The common performance measurement habits are bad habits

Just about all the things that most of us currently do to choose, implement and use performance measures are the same things we’ve always done, and the same things that everyone else does. But common practice doesn’t mean good practice. Our struggles with performance measurement are caused by common practices that are actually bad habits. We’ve looked briefly at some of these bad habits in previous chapters. In Chapter 1 we took note of things that are called performance measures but really aren’t. And, in Chapter 2 we looked at symptoms of a broken performance measurement process. In those discussions I alluded to several common problems that we’ll now look at systematically. Here are eight of what I consider the most insidious of bad performance measurement habits.

Bad Habit #1: Using measures to judge

‘Whenever there is fear, you will get the wrong figures.’

— W. Edwards Deming

Measuring people, or using measures to judge or assess how people are performing, is a bad habit because it makes people fear what the measures will show. The consequences are especially bad when measures are used for rewarding or punishing people. People will make excuses, fudge the figures and sweep problems under the rug to protect themselves. Measuring people, or using measures to judge people, does not encourage true performance improvement.

An example from when I worked for the railways was people leaving cancelled trains out of the calculation of the On Time Running measure so that the measure would look okay. People would even cancel train services that were running late enough to affect their On Time Running measure.

In call centres, Average Call Handling Time is a common measure. What do you think a call centre operator is going to focus on when they talk to a customer if the target is to keep average call durations under three minutes? The primary focus will probably not be helping the customer with their problem, especially if it looks like it will take longer than three minutes!

We don’t need to measure people’s performance. Managers need to coach and facilitate people’s performance, but measuring it only causes fear. And we don’t need to measure people in order to reward them financially either. Dan Pink⁷ is just one of many contemporary experts who credibly discusses the problems with the outdated method of using money as a motivator. One problem is that people are more intrinsically motivated by fulfilling a purpose than they are by increasing a company’s profits or their own. Another is that people want their work to be creative and self-directed; as soon as we start measuring them, it turns their work into drudgery. Humans are not economic robots. Let them use measures to creatively improve the work they do, not as rods to flog their backs.

Bad Habit #2: Using weasel words

‘There have been signs of decay in the language of politics and academia for years, but the direst symptoms are in business; and the curse has spread  .  .  .’

— Don Watson⁸

Using management jargon, clichés and other infamously wishy-washy words to write our goals or objectives is a bad habit that people joke about but still accept as the way things are done. The following are becoming well known as ‘weasel words’:

adaptive | growth | quality
balanced | holistic | reliability
capacity | inclusive | sustainable
dynamic | key | transparent
effective | leveraged | utilised
foster | needs | value
empowered | outcomes | well-being
fit for purpose | productivity |

Using weasel words is a bad habit because these words have multiple undefined meanings—(or no clear meaning at all), making goals and objectives nearly impossible to measure. Imagine trying to quickly find meaningful measures for goals like these:

Provide efficient, unique, unbiased, and responsive, high-quality support (from a military organisation)

Strengthen student engagement and learning outcomes by enhancing student support and intervention services (from a government education department)

Be a leader in articulating and characterizing the dynamic system of scholarly communication (from a library association)

I’ll bet you can’t quickly find a measure that would determine whether any of these goals are achieved. And I’ll bet many of you grimaced because your organisation or business has goals or objectives like these too.

Bad Habit #3: Brainstorming KPIs

‘If it matters at all, it is detectable/observable. If it is detectable, it can be detected as an amount (or range of possible amounts). If it can be detected as a range of possible amounts, it can be measured.’

—Douglas Hubbard⁹

Brainstorming ideas for performance measures and then voting on them is a bad habit because it produces lists of measures that are as useful as a screen door on a submarine. Typically, brainstorming results in too many suggested measures and no other way to work out which ones to choose than by a popularity contest. Often the ideas that are proposed aren’t even performance measures, and this little fact goes unnoticed by just about everyone. Later we all wonder why we can’t implement the measures, or why those measures don’t seem to be very useful.

Imagine brainstorming measures for innovation: ‘Okay, everyone: what are some measures for innovation?’ The first suggestion is ‘Innovation Survey’. But that’s not a measure, it’s a data collection method. Another suggestion is ‘Innovation Training’. Again, that’s not a performance measure, it’s an action. The next suggestion is ‘Number of ideas generated’. It’s a measure, but a very trivial one that gives little evidence of the results we likely want from innovation. Someone suggests ‘Innovation Success Rate’. It sounds fabulous, but no one can describe exactly how to calculate it.

Brainstorming is a creativity tool, not a technique for selecting performance measures.

Bad Habit #4: Asking people to simply sign off

‘[M]easurement is not something that should be reserved for the select few; in a high-performing organisation, measurement is everybody’s job.’

—Dean Spitzer¹⁰

Saving people the pain of getting involved in measurement by doing it for them and then emailing a measure report to get their sign-off is a bad habit. It’s bad because we don’t get true buy-in or ownership when they are simply asked to accept a list of measures that others chose or designed. People will take a cursory look at what we produced, make some assumptions, and tick their sign-off. Then later on they will debate the measures, complain about data quality problems, and make excuses for not hitting targets.

A few years ago I was invited to submit a proposal to help an agency design a set of performance measures on behalf of several other government agencies. They declined my proposal because they felt it required too much time and input from their stakeholders. A few years later I was invited by three of those stakeholders to help them come up with some performance measures that suited them because they didn’t fully buy into what the original agency ended up producing.

Bad Habit #5: Rushing to data and dashboards

‘Today, everybody wants a dashboard. Like many newcomers to the technology scene, dashboards are sexy. Software vendors work hard to make their dashboards shimmy with sex appeal. They taunt, “You don’t want to be the only company in your neighborhood without one, do you?”’

—Stephen Few¹¹

Handing over poorly detailed performance measures to analysts and expecting them to collect and analyse the relevant data—or to dashboard teams to plug into a performance dashboard—is a bad habit. It’s bad because what usually ends up in the performance report or dashboard is not what the measure owners intended. Sometimes that’s obvious to the measure owners, and they’ll complain that the dashboard isn’t useful or trustworthy. But sometimes it’s not obvious to them, and they go right ahead and make decisions with the wrong information.

How many different ways do you think you can calculate a measure of on-time performance? If that’s all the operations manager asks for then they are going to end up with a range of possible values, each based on different types of statistics (such as a percentage or an average), different sources of data, and different rules about scope. The same goes for many other seemingly innocent performance measures including customer satisfaction, employee engagement, return on investment and even profit. These could all be calculated in numerous and very different ways.

Bad Habit #6: Using reports to CYA

‘The annual report is a prime example of a document of declaration. It is a marketing document, and CEO letters and interviews that are part of the annual report provide sound bites that cannot be verified. The abuse of the annual report through exaggeration and marketing hype has almost rendered this expensive document a useless artifact.’

—A. Larry Elliot & Richard J. Schroth¹²

Filling performance reports with lots of details—tables of numbers, assorted graphs, progress reports, action milestones and reasons (excuses) for not hitting targets—is a bad habit. Usually this kind of padding happens because people are worried that managers will ask questions about unacceptable performance, so the report preparer adopts the CYA (cover your arse) principle. When bad habit #1 (using measures to judge) is alive and well, the CYA-style of performance report is made as thick as possible. Detail might sound potentially useful, but reports like these fail to answer the three fundamental questions that would enable users of the reports to take the right action to improve performance:

1. What is performance doing relative to target?

2. Why is it doing that?

3. What do we need to do to move it closer to target?

The questions that CYA-style reports do answer are more about program management or statistical status reports:

How was performance for last month?

Who did what during last month?

What are all the measures and analyses related to what we did?

What are the projects we worked on?

How many milestones did we hit?

Why didn’t we hit all the planned milestones?

The focus is in the weeds, so no-one is managing the landscape.

Bad Habit #7: Comparing this month to last month

‘Thus we have the two mistakes that we can make when we attempt to use data. Either we can interpret noise as a signal and chase after explanations that do not exist. Or, we can interpret a signal as a bit of noise and fail to learn that which could be learned.’

—Donald Wheeler¹³

Deciding whether performance is getting better or worse based on what this month did compared to last month is one of the most insidious of the bad habits. And it’s more invisible to people than any other. It’s bad because it ignores the fact that all things naturally vary. What makes last month an appropriate baseline to evaluate this month’s performance? What if last month was unusually good, or abnormally bad? Do we really think that every variation is a signal of change? All things vary routinely to some extent, even in a stable and unchanging system. The real question should be: how can we tell when the pattern of routine variation has changed?

If we set arbitrary cut-offs for percentage-difference comparisons between this month and last month, how do we know whether the cut-off we chose is smaller or larger than the routine variation of the measure? We don’t know unless we’ve measured the routine variation of each of our performance measures. If we use statistical techniques, we have to also use statistical thinking. Comparing this month to last month, target, year-to-date or the same month last year are all statistically flawed methods for attempting to get the truth out of our measures.

Bad Habit #8: Using educating, resourcing or funding as improvement initiatives

‘As organizations begin to accurately specify value, identify the entire value stream, make the value-creating steps for specific products flow continuously, and let customers pull value from the enterprise, something very odd begins to happen. It dawns on those involved that there is no end to the process of reducing effort, time, space, costs, and mistakes while offering a product which is ever more nearly what the customer actually wants.’

—James P. Womack & Daniel T. Jones¹⁴

Fixing performance problems by educating people, increasing staffing levels or seeking more funding/a bigger budget is a bad habit. It’s bad because all these methods do is compensate for sub-par performance rather than fix the underlying causes of it. This approach assumes the work processes, policies or systems designed to deliver the results we’re measuring are working as well as they can. It assumes the cause is ‘out there’, so rather than changing something we are doing ‘in here’ we need more resources to cope with the problem.

I listened recently to a radio program about a researcher trying to find a way to increase the number of patients in rural communities who stick with long-term medical treatment. Educating the patients about the importance of continuing their treatment wouldn’t have made a difference. Neither would have throwing more government funding at rural health programs. And neither would have increasing the staffing levels in rural health clinics. Why? Because the root cause was the discomfort rural men have with being treated by female health practitioners, who happen to make up the vast majority of health practitioners willing to work in rural communities. Before we can improve performance problems, we need to understand their root causes. When we fix the root causes instead of compensating for them, we generally get a bigger and longer-lasting result for a smaller investment.

What can replace these bad habits?

Getting better at performance measurement—and getting performance measures that truly help us reach targets, fulfil our organisation’s mission and achieve its vision—isn’t just about stopping those bad habits. It’s about replacing them with good practices that work. Discovering both the bad habits and the fact that better practices do exist is what will get our colleagues curious about trying PuMP with us.

Our colleagues’ outdated beliefs about performance measurement will change as a natural consequence of experiencing a PuMP implementation. Experiencing PuMP implementations several times over and getting great measures that truly improve performance inspires people to change their bad habits. The first PuMP Pilot entails showing them what the existing bad habits are, and holding out the promise that if we change those habits our performance measures will be vastly more meaningful and useful. And when our performance measures are vastly more meaningful and useful, we will get vastly more intrinsic enjoyment, satisfaction and pride from our work—and the difference we’ll be making in the world.

The first PuMP meeting is also about establishing a foundation for buy-in. Once we have a Measures Team, we’ve taken the first step toward building a foundation of buy-in to performance measurement. But we’ve still got a long way to go. Committing time to attend a performance measure meeting is a far cry from seeing performance measurement as an essential and valued part of the ‘real work’. So our first PuMP meeting needs to fan the flames of buy-in so at the very least the Measures Team members feel curious—ideally, excited—about coming to the second meeting. The PuMP Diagnostic fans the flames of curiosity in two ways. Firstly it guides a dialogue that explores what good performance measurement practices actually are. Secondly it gives them a baseline measure of how mature their current approach to performance measurement is, and inspires them to make it better.

Step 1 of the PuMP Blueprint is designed to warm up our colleagues—specifically those who will work with us as our Measures Team—by taking them through the PuMP Diagnostic Discussion. This is a dialogue about what good performance measurement practices are, and how our current practices are positioned relative to these. It’s an easy and engaging way to introduce the Measures Team to PuMP as a collection of practices to replace those bad performance measurement habits:

Step 1.1: Discuss the diagnostic criteria

Step 1.2: Capture and summarise ratings

Step 1.3: Discuss how PuMP will lift the ratings

The 3 steps to warm up the Measures Team

Step 1.1: Discuss the diagnostic criteria

While the PuMP Diagnostic is a lot more structured than the PuMP Mind Map activity described in Chapter 2, it still guides a good discussion about performance measurement problems and how they can be solved.

The PuMP Diagnostic is about discovering how our entire existing performance measurement process is working, not just the selection of our performance measures. It includes looking at how performance data is collected, stored and analysed to produce performance information, as well as how that performance information is presented, interpreted and applied to manage performance improvement.

The PuMP Diagnostic questions gauge how well our organisations meet the critical attributes of ideal performance measurement practice. The criteria are laid out in the form of scales that describe bad performance measurement practice (including those bad habits discussed earlier) at one end, good practice at the other, and a continuum between the two. The idea is that we will locate where our organisations or teams currently sit on that continuum, using a 10-point scale.

The diagnostic criteria are clustered under five headings that generally align with the PuMP Blueprint, and you can see the complete list of criteria in Diagnostics 1–5.

With the Measures Team, we discuss each question in the Diagnostic in turn. We check whether everyone understands the question. We ask whether team members have considered each question in the past. We discuss where we all feel our organisation or team currently resides between the ‘undesirable’ and ‘desirable’ practices described. We can aim for consensus, or we can let everyone select their own rating.

Step 1.2: Capture and summarise the ratings

We can collect everyone’s ratings either individually for each criterion in the Diagnostic, or we can try to reach a single consensus rating. If we do this in a spreadsheet (there’s a to link where you can download a ready-made one at the end of this chapter), either the averaged individual ratings or the consensus rating will be summarised for each question and each of the five parts of the Diagnostic.

By doing this we can see at a glance where the existing process is stronger, and where we need to focus more on improving our performance measurement process. This can give us additional information to assess how long each step of the PuMP Blueprint might take to implement. Recording the ratings in the Diagnostic spreadsheet also creates a benchmark so that we can gauge how much better the team is getting at performance measurement. Some facilitators schedule a second PuMP Diagnostic Discussion at the end of the PuMP Pilot to look at how the ratings have changed.

DIAGNOSTIC 1. How measurable is our strategy?
Diagnostic 1

DIAGNOSTIC 2. How meaningful are our measures?
Diagnostic 2

DIAGNOSTIC 3. How well are our performance measures implemented?
Diagnostic 3

DIAGNOSTIC 4. How useful and usable are our performance reports?
Diagnostic 4

DIAGNOSTIC 5. How well are performance measures improving performance?
Diagnostic 5

Step 1.3: Discuss how PuMP will lift the ratings

Here we share with our teams how the PuMP Blueprint steps will directly replace the ‘undesirable’ practices with the ‘desirable’ practices. Having gone through the Diagnostic, and hearing the promise PuMP holds for improving the existing process, the Measures Team might be a little more curious and even a little excited about starting Step 2 in the PuMP Blueprint at the next meeting. And that’s where the real fun begins.

Take Action: Engage the Measures Team

Get the Measures Team together for an introductory meeting to explore a fresh approach to performance measurement, using the PuMP Diagnostic Discussion tool.

Go to the book’s website (www.pumpblueprintbook.com) and download the PuMP Diagnostic instructions and template.

Chapter Summary

Buy-in to performance measurement (or engagement) is one of those non-negotiable ingredients for getting the kinds of measures that transform organisational performance. There’s a social dimension to performance measurement and management that can’t be denied, so nurturing buy-in from the very beginning is the only way to go.

Starting from the experiences, beliefs and attitudes our colleagues have right now about performance measurement is a natural way to introduce a different approach. We can then frame that new approach in the context of replacing the bad habits of performance measurement with practices that solve our problems and eliminate our struggles.

Measuring the existing performance measurement approach by using the PuMP Diagnostic is a great way to objectively benchmark and see progress in improving our performance measurement process.

Now it’s time for the Measures Team to make their goals measurable and tease out the performance results from those goals, as the focus for what we will measure.

5

Step 2: Mapping Measurable Results

‘Words are pale shadows of forgotten names. As names have power, words have power. Words can light fires in the minds of men. Words can wring tears from the hardest hearts.’

— Patrick Rothfuss, The Name of the Wind

‘But if thought corrupts language, language can also corrupt thought.’

— George Orwell, 1984

When people say they struggle with finding meaningful performance measures, I almost always find their problem is not in how they look for meaningful measures but rather in articulating what they want those measures for. What is being articulated is usually their strategy—Key Result Areas, Critical Success Factors, goals, objectives, or whatever names they give to their statements about what the performance priorities are supposed to be right now.

So, more often than not, the best first question to ask when we’re on a quest for better performance measures is not ‘How do we measure that?’, but rather ‘How measurable is our strategy?’

Before we can discuss this question meaningfully, we need some further definitions because we don’t all use strategy terminology the same way. Let’s define some key strategy terms using Chokolaide, the fictional not-for-profit organisation I mentioned in Chapter 1 when discussing terminology as our example. I am in no way proposing that what follows is the correct definition of these terms. My goal is just for us to agree on a meaning to understand what I’m saying in this chapter.

Why is most strategy immeasurable?

One problem with the strategy in most organisations and companies is that in its sanitised, word-smithed and published form it’s not measurable. There are several reasons for this:

Weasel words and vague ideals, including ambiguous terms such as ‘best practice’, ‘optimised’, ‘enhanced’, ‘effective’, ‘efficient’, ‘productive’, ‘responsive’, and ‘sustainable’.

Action orientation, which means the strategy is expressed as a collection of initiatives or projects.

Multi-focus goals, like aiming to have ‘responsive, efficient and high-quality services’.

Confusion with business as usual (e.g. planning to ‘continue our current level of performance’).

Lack of rigour, logic and strong cause-effect and systemic relationships that tie the strategy together.

Each of these flaws appears in varying degrees in just about every strategic plan I’ve seen. The prevalence of these problems is a sign that we need to change how we write our strategies. We need to do this not just to make it easier for us to meaningfully measure our performance in carrying out strategy, but also to make it easier to explain our strategy and have others understand how they can contribute to executing and achieving it.

If you and your colleagues are struggling to find meaningful performance measures for your goals or objectives, take a closer look at how they’re written. Let’s run through how we can assess the measurability of a strategy.

Weasel words and vague ideals

Use of ‘weasely’ language is probably biggest reason that strategies are immeasurable. Wikipedia explains what ‘weasel words’ are:

The expression ‘weasel word’ derives apparently from the egg-eating habits of weasels. Regardless of whether weasels in fact suck eggs, a belief that they do implies an egg shell devoid of its contents. Thus, words or claims that turn out to be empty upon analysis are known as ‘weasel words’.

The expression first appeared in Stewart Chaplin’s short story ‘Stained Glass Political Platform’ (published in 1900 in The Century Magazine), in which they were referred to as ‘words that suck the life out of the words next to them, just as a weasel sucks the egg and leaves the shell’. Theodore Roosevelt attributed the term to Dave Sewall, claiming that Sewall used the term in a private conversation in 1879. Winston Churchill wrote: ‘The reserve of modern assertions is sometimes pushed to extremes, in which the fear of being contradicted leads the writer to strip himself of almost all sense and meaning’.  .  .  .

In the political sphere, this type of language is used to ‘spin’ or alter the public’s perception of an issue. In 1916, Theodore Roosevelt argued that ‘one of our defects as a nation is a tendency to use  .  .  .  “weasel words”; when one “weasel word” is used  .  .  .  after another there is nothing left’.¹⁵

My first exposure to the term ‘weasel words’ was thanks to Don Watson, an Australian political speech writer and author of many books, including two about weasel words: Death Sentence: The Decay of Public Language¹⁶ and Watson’s Dictionary of Weasel Words.¹⁷

The vagueness and ambiguity introduced by weasel words in our strategies are major reasons for our struggles to find meaningful performance measures that align with those strategies and allow us to quantify how well we are executing them. Look at any strategic plan, perhaps even your own, and chances are you’ll see plenty of weasel word such as:

accelerate | brand (image) | consultative
accessibility | capability | continuing
accountability | capacity | core
active | centrality | delivery
adaptive | challenges | demonstrated
advocating | change | deployable
alignment | client-driven | dis-established
balanced | collaborative | diversity
barriers | compelling | drive
basic | competence | dynamic
benchmarked | competitive | effective
benefits | connectedness | efficient
best practice | considered | embedded
empowered | interdependent | responsibility
enablers | interface | responsive
end-to-end | issues | rich
end-user | key | rigorous
evidence-based | lean | roadmap
focus | leveraged | robust
foster | liveability | significant
empowered | mandated | solution
engaged | needs | special
enhancements | network | standards
excellence | opportunities | strategically
fit for purpose | optimised | streamlined
flexibility | outcomes | strengthened
flourish | outputs | strong
forward-looking | oversighting | successful
fundamental | ownership | supported
going forward | pathways | sustainable
gold-plating | participation | synergies
governance | performance | targeted
ground-breaking | priority-driven | transformation
growth | proactive | translational
harness | processes | transparent
harvested | productivity | underpinning
high-value | products | uniqueness
holistic | progression | utilised
implications | progressive | value
improvements | quality | value-added
inclusive | reanimate | vanilla solutions
initiation | recognition | vibrant
innovative | re-engerise | well-being
input | reform | winning
integration | reliability | world-class
integrity | renewal |

This is just a sample of the glut of empty words that sound important but fail to say anything at all, or at least to speak of anything that can be verified or measured in the real world.

Here is a real-life example of weasel words in action. Three of the objectives in the Strategic Plan of the Australian Department of the Premier and Cabinet (DPC) are:

‘Cabinet and Cabinet Committee decision-making supported by efficient and effective processes’

‘Supporting the Premier by maximising opportunities for [the State]’

‘DPC capacity supports our priorities’.

These objectives are composed almost entirely of words that are not verifiable, and could mean different things to different people. What does a ‘supported decision-making process’ look like? What kinds of processes will do that supporting, and what aspects of those processes need to be more efficient and more effective? What kind of support does the Premier need to maximise opportunities for the State? What kinds of opportunities need to be maximised? What is meant by ‘capacity’? Is it the level of knowledge or skill, the size of their budget, having enough people, or something else?

The lack of specificity in weasel words renders a goal immeasurable. In these examples, if we knew that ‘capacity’ was about having enough people, then we’d have some chance of finding a meaningful way to measure it. It would probably be something like the availability of the right number of staff with the right experience or skill to fulfil a Cabinet request. But we have no way of knowing if that is what the objective means.

Often people don’t return to a strategy to select measures for some weeks or even months after writing it. When they do, even those who were actively involved in the writing will say they can’t remember what they meant. This is a sure sign that the strategy contains weasely or vague language. The chances of meaningfully measuring objectives laden with weasel words are small to none.

Vague ideals are a subset of the weasel words category, and refer to goals or final targets expressed in language so ambiguous that we will never know when or even if we achieve them. Often these vague ideals have a patriotic ‘motherhood and apple pie’ quality, using terms like ‘world-class’, ‘best practice’, and ‘optimised’ to refer to a state of perfection we hope to achieve.

Here is an example of a vague ideal from a public utility: ‘We will be a world’s best-practice energy provider by 2010.’

This ideal is not measurable in a meaningful way. We’d need to find a method to quantify what constitutes the ‘world’s best-practice’ in the energy sector, but in which dimensions specifically? In reality, even different organisations within the same sector have strengths in different areas. It is unlikely that one organisation would excel above the rest in all dimensions. Vague ideals like this one express such a ridiculously lofty goal as to be impractical and demotivating.

Here are some more examples of goals and objectives that fail to give sufficient focus for measurement or even execution:

‘Deliver state-of-the-art shared services (HR, FM, GS, BLST)’ (from a military organisation).

‘Support general practice to facilitate and encourage optimal and culturally appropriate health care of indigenous persons’ (from a health promotion charity).

‘To continue to be one of the best cancer research centres in the world’ (from a cancer research organisation).

‘To deliver a world-class research, learning and education network, and supporting services’ (from a research and education network).

‘Ensure we implement best practice and high standards of environmental sustainability throughout our operations’ (from a nature reserve management organisation).

‘State-of-the-art’, ‘optimal’, ‘best’, ‘world-class’, ‘best practice’, ‘enhanced’, ‘stellar’, ‘maximised’, ‘minimised’ and so on are all vague ideals that will distract us from the right things to measure.

Goals or objectives that express vague ideals might aim to inspire people. But in an organisation that doesn’t have a track record of excelling, even on a small scale, goals such as these only foster cynicism in the workforce. Being ‘the best’ can require an inordinate volume of resources, and is such a tenuous and fragile position to strive for because we fail if just one other organisation beats us to it. Aiming to be ‘world’s best-practice in the energy sector’ might be symptomatic of an ego problem, where the company’s executives are excited by the idea of being the best before they’ve even managed to consistently meet the current basic standards and expectations of their stakeholders.

Often the use of vague ideals distracts people from the real performance result that a goal or objective aims to produce. People ask, ‘How do we measure best practice’? The answer is: we don’t. Vague ideals are really qualitative targets, indicating the direction or level of performance for which an organisation wants to aim. We don’t measure targets, we measure results. And then we set targets for the measures, so we can monitor change over time and do what it takes to move the measure’s actual values closer and closer to the target value.

When goals or objectives contain weasel words and vague ideals, we can’t find meaningful performance measures for those goals until we define specifically what we mean by these ambiguous terms. A strategy’s purpose is to let the members of the organisation know where they’re heading and what paths they’ll take to get there. If people can’t interpret the strategy, they’ll be heading anywhere or nowhere, and most definitely not all in the same direction to the right destination. We won’t have a clear and measurable strategy unless we use rich, descriptive words to articulate the differences or results that our goals or objectives are meant to achieve. We’ll see later in this chapter how we can use measurable descriptive language to ‘de-weasel’ our goals.

Most people who use vague language in their strategies are simply caught in the fast-flowing tide of weasel words without realising it. But sometimes weasel words are used deliberately to leave things vague and allow for flexible interpretation of actual results in a post-hoc analysis of whether strategy has been successful. That’s the same as saying our direction or strategy is open for interpretation, and any interpretation will do. That is not a clear direction.

Action orientation

We have difficulty finding meaningful measures for our strategies when they are made up of goals or objectives that are actions, not results. Here’s an example from a government Office of Personnel Management’s ‘Professional Development’ key result area: ‘Issue a comprehensive catalogue of Federal Government-offered professional development programs’. The problem is that this goal is an action: it simply states that something will be done, in this case a catalogue of training programs will be issued. Meaningful measures track results: the impact, outcome or difference that we’re trying to reach or attain. Our actions are how we decide to attain a result, but they are not an end in themselves. To measure whether our example goal was completed, we’d simply look at whether the catalogue was issued or not. But the fact it was published does not tell us how well the catalogue actually contributed to the professional development of personnel.

Just because we choose a specific action to improve performance doesn’t mean that completing that action equates to improved performance. What result could this Office of Personnel Management have really wanted to achieve by issuing a comprehensive catalogue of professional development programs? What would they see if the catalogue was successful? Perhaps the result they sought was to make it easier for government agencies to find appropriate courses for their staff. That particular result is more meaningful to measure because it focuses on the purpose of the action, and would give feedback allowing us to test and tune the action, to make sure it stays on track to achieve that result. For this goal of the Office of Personnel Management to be measurable in a meaningful way, we’d first have to find out what result they wanted from issuing the professional development programs catalogue. The measure would be designed to evaluate the evidence that the result was achieved.

Here are some more examples of goals that are written as actions and lack a clear indication of the intended results:

‘To invest in core digital and estate infrastructure’ (from a museum).

‘Create a network of priority bus corridors’ (from a public transport organisation).

‘Develop tools and platforms to analyse data’ (from an information technology branch).

‘Completion of the identification and baseline information about our Natural Assets incorporating options for financing the long-term management and maintenance of these natural assets’ (from a municipality).

‘Improving our stakeholder engagement processes’ (from a local government association).

A strategy that is expressed as a collection of actions—initiatives, projects or milestones—is only about what we’re going to do, not about the difference we want to make through our actions. Measuring actions encourages more action, faster action, but not necessarily the right action. By contrast, measuring results gives us feedback that we can use to determine whether our actions have the effect we intended, or whether to make a specific change or improvement in our results. Sure, we need to monitor how well we’re implementing the projects and initiatives we’ve chosen to pursue to improve our performance. But that’s called project management or program management, not performance management. Evidence of the performance of an organisation is in the results it produces not how many activities it carries out. So a strategy that is little more than a collection of initiatives is a strategy that risks achieving little more than consuming resources.

Multi-focus goals

It’s very important to have a concise, focused strategy that clearly defines the priorities for helping the organisation succeed in its current environment. This means we don’t want a corporate strategy that has dozens of critical success factors, key result areas, goals or objectives. If everything matters, then nothing will be achieved. Although we mostly know this, there’s a little trick we sometimes play to squeeze in a few more of those things that matter without making the strategy look any bigger. We join a few goals together into one statement that looks like a single goal, but isn’t. I call this a ‘multi-focus goal’.

Here’s an example objective from a local government agency: ‘[Our Council] will increase its responsiveness, efficiency and effectiveness in delivering high-quality services’.

Aside from the abundant use of weasel words (‘responsiveness’, ‘efficiency’, ‘effectiveness’, ‘high-quality’), notice how many specific results this objective implies. One result is the responsiveness of the Council’s services, another is the efficiency of those services, another is the effectiveness of those services, and yet another is the quality of the services. How many services are there? Just that first sentence in the objective already implies a bare minimum of four results.

And because these are weasel words, each one possibly implies several results as well, particularly the word ‘quality’, which usually requires the identification of the specific attributes of quality that are most important to customers or clients of each of the services. I feel exhausted just thinking about trying to measure an objective like this one.

From a bank: ‘Achieve the highest levels of operational efficiency and effectiveness in all that we do’.

The dual weasel words ‘efficiency’ and ‘effectiveness’ are one multi-focus element of this statement. Efficiency in performance is different from effectiveness, so at the very least we’d need to be aware that this statement requires us to measure each of these qualities. There’s no single measure that can meaningfully assess whether something is both effective and efficient.

The other multi-focus element of this statement is the innocent little phrase ‘in all that we do’. There are so many different services, processes and functions in the bank that this phrase encompasses a multitude of goals, with no focus at all. The only way to know whether such a goal is achieved is to delve into all the sub-goals that make it up. All these sub-goals will very likely target different types of results in different parts of the organisation and would need different measures. Efficiency of the tellers in the bank’s branches is different from the efficiency of the loan applications process, and different again from the bank’s investment process. There’s no single measure that will tell us the performance of everything—not meaningfully, anyway.

The problems with the formulation of this bank’s objective are symptomatic of a lack of meaningful analysis during the organisation’s strategy development process. Because of this lack of analysis, the bank failed to develop clear priorities. Attempting to measure a goal like this would send the organisation into measurement overwhelm, rendering them unable to measure—or achieve—anything at all because when everything matters, nothing matters. The best way forward when goals are this vague, and focused on so many things, is to revisit the strategy development process and become ruthless about focus.

Here are some more examples of multi-focus goals:

‘[We] will have a range of powerful and effective alliances to help shape and promote the various new models appropriate for different disciplines and communities’ (from a library association).

‘Provide efficient, unique, unbiased and responsive, high-quality support’.

‘Proactively attract, support and develop high-quality staff’ (from a university).

‘We assist people to deal with all forms of illness and to live a satisfying life where they can contribute to their community’ (from a state government).

‘Support [general practitioners] to provide early intervention and optimal care and contribute to the achievement of the best possible health outcomes for patients with mental health disorders, and assist in the reduction of the impact of mental disorder on individuals, families and communities, in collaboration with other mental health services as appropriate’ (from a general practitioners’ network).

Smooshing several goals together doesn’t change the number of results we’re trying to achieve, or the number of measures we’ll need to monitor the full impact of the goal. The more adjectives and nouns we have in our goal or objective statement, the more implied results there are, and the more measures we’ll need if we’re serious about fully achieving that goal or objective. We most certainly will not find a single measure that gives useful and usable evidence of a collection of specific results. And we should not even think about trying to create a single index by ‘rolling up’ all the detailed performance measures. Rolled-up indexes hide important signals of change in performance and keep decision makers too far removed from the real, tangible results they are responsible for creating. Each performance result needs its own measure—sometimes even two or three measures—to provide actionable information.

Confusion with business as usual

Goals that start out with phrases like ‘Continue to.  .  .’, ‘Maintain our.  .  .’ or ’Strive to.  .  .’ aren’t really goals at all. They state the obvious, and say little more than ‘We’re going to keep going as we have been’. Strategy is about change, improvement and getting better results than we have gotten so far. We won’t get better results by continuing to do the same things. (Einstein’s definition of insanity is ‘doing the same thing over and over again and expecting different results’.)

One of the goals for a health and welfare institute’s key result area of protecting privacy is, ‘Continue to refuse requests for information when they threaten privacy or do not follow the conditions set by those supplying the base data’. Amidst their other goals, which were focused on change and improvement, this goal really serves no purpose strong enough to take up valuable space in the strategic plan. They wouldn’t be measuring it to aim for improvement. That’s not to say they shouldn’t measure how well they do protect the privacy of their data. But if they were to measure how well they protect the privacy of their data without setting a target for improvement, then what they would be measuring is business as usual, not strategy.

Here are some more examples of goals that read too much like business as usual:

‘[We] will continue to be a leader in advocacy and educational efforts within [local] and international research and educational communities’ (from a library association).

‘Continue to provide recycling and organic waste options in staff areas including kitchens’ (from the environment plan for a public record office).

‘Continue to provide up-to-date equipment and convenient, easy user access to online information resources and services sufficient to support the college’s programs at all campuses and instructional sites’ (from a community college).

‘Maintain the level of brand awareness in existing markets’ (from a sporting association).

‘Maintain the level of safety and security currently enjoyed by the community’ (from a local government organisation).

Certainly we can (and should) measure the business-as-usual results. But not all of them. And certainly not all of our business-as-usual measures need targets and improvement initiatives. Often we’re just measuring business as usual to be sure we get quick feedback about any critical exceptions to how things should be performing. Ideally our business-as-usual measures will be simply and routinely tracked and reported, and we’ll dwell on them only when they flag a problem. Most of our performance management time will be spent on our strategy related performance measures because that’s where we already know we need to make improvements.

Lack of rigour and logic

People who are inexperienced at developing strategy, or who are not using a deliberate framework to guide the design of their strategy (e.g. EFQM, Baldrige, Performance Prism, Balanced Scorecard), often end up with a strategy that just doesn’t hold together. And when a strategy doesn’t hold together, attempts to reach targets in one key result area can end up sabotaging efforts to achieve targets in another key result area.

Back in my railway years, I heard of a tragic situation that resulted in part from this very problem. For the railways, on-time running of trains was probably the single most discussed and managed performance result. Train safety (e.g. keeping the trains on the tracks) was important too, but in designing targets for performance measures the organisation didn’t make clear the relationship between on-time running and train safety. A situation arose in which the CEO personally called a train driver to make sure that he ran a particular service on time because the customer was irate about recent delays. In response, to save time before departure the train driver reportedly omitted a routine safety check: walking the length of the train to double-check all the brake couplings. As fate would have it there was a problem with the brake couplings, and that particular service ended up derailing on a bridge because the driver couldn’t slow the train down. This story starkly illustrates that performance results (and their measures) should not be individually traded off against each other but rather managed as a coordinated whole.

A strategy that doesn’t hold together usually hasn’t been disseminated very sensibly throughout the organisation either. As a result, people are given goals or objectives that make little sense to them simply because those objectives are in the corporate plan. There’s no sense at all in cascading a corporate objective to ‘reduce Lost Time Injuries by 8%’ to the Accounts department where the injury risk is little more than a paper cut.

The typical approach executive teams use to cascade/roll out their strategic direction is to produce a clear set of goals, objectives, critical success factors, or a scorecard, and then get each departmental or functional manager to customise them for their department. I call this the ‘Mini-Me’ syndrome, and if you’ve seen any of the Austin Powers movies with actor Mike Myers you’ll know what I mean. (And if you haven’t, read the footnote.¹⁸)

In a typical planning approach, the first phase has the organisation’s senior leadership team designing and expressing a strategic direction using a framework of some kind, based on a business environment analysis methodology such as a SWOT analysis. The organiation’s strengths, weaknesses, opportunities and threats are identified and analysed to reveal a few specific themes that become the framework of the new strategy. As I mentioned earlier, these themes might be referred to as key result areas, critical success factors or strategic themes. The team develops strategic goals or objectives within each part of this strategic framework, along with a set of key performance indicators. The result could look something like the excerpt in Table 1.

The next phase in this traditional planning approach is often to communicate the strategy to the rest of the organisation, encouraging the next layer of management to translate it into a tactical or operational-level strategy. But what usually happens is the functional managers (of business units, departments or whatever the divisions of the organisation are called) create their own sets of goals aimed at contributing to the achievement of the organisation’s strategic goals. The result can be problematic at best because not every organisational goal applies to individual departments, and not every department has the scope or capability to support all of the organisational goals.

TABLE 1. Excerpt of a corporate strategy
TABLE 1. Excerpt of a corporate strategy

For example, imagine that the Corporate Services Department of an organisation (the part that manages the internal support processes like purchasing, payroll, and information services) is trying to translate the corporate strategy into the department’s own operational strategy. Let’s say that one organisational goal is ‘customer advocacy’. The Corporate Services Department decides this goal is not relevant to them because their customers can only ever be other internal departments of the organisation. By contrast, the ‘customer satisfaction’ goal is important to them, so they might establish a goal related to internal customer satisfaction. But then what will they do for the profitability goal? Perhaps come up with a goal related to budget performance. And what about the goal of reducing costs? They can’t really reduce their labour costs because the rest of the organisation already puts more demand on them than they can effectively meet, so they establish an operational goal of reducing their consumables expenditure.

What’s wrong with this picture? The Corporate Service Department is trying to create a Mini-Me version of the organisation’s overall strategy, regardless of whether the organisation’s goals are applicable to the department, will compromise the department’s ability to do its job, and whether the department has other goals specific to its function that should get priority.

The process of applying organisational strategy to individual departments can get even more specific. Let’s say a corporate target for downsizing (or ‘head count reduction’, or ‘right sizing’, or any other term that means ‘getting rid of employees’) is 10%. In the Mini-Me approach, every department could be expected to reduce its size by 10%, regardless of whether the department has the scope to downsize by 30% or is already struggling with the number of people it has now. Or a corporate safety goal could be to reduce the Lost Time Injury Frequency Rate (LTIFR) to 8, and having every department expecting to achieve an LTIFR of 8 whether its starting point is 9 or 42. Needless to say, cascading targets like this causes all kinds of chaos, sub-optimisation, cynicism, wasted resources and missed opportunities. And more often than not, the corporate target isn’t achieved. That’s why we need to test the rigour and logic in our strategy design, which has everything to do with how that strategy is disseminated throughout the organisation.

If we don’t cascade strategy well, we’ll be dealing with a range of problems. One of the typical problems is discovering there are one or more goals in the corporate scorecard that a particular department can’t sensibly adopt, or even adapt. For example, many departments that don’t have external customers have no use for goals about customer loyalty or customer referrals. Nor do they have any use for a goal about profitability. For departments with minimal resources that are already struggling to cope, further cost cutting just because it’s a strategic goal can compromise performance to the point that it undermines the achievement of other goals.

Another typical problem is that when a departmental team sits down to develop their own operational strategy, they struggle to relate the long-range, all-encompassing corporate goals to what they can do and influence in the shorter term. For example, consider a corporate goal of ‘enhanced corporate image’. How does an individual department, particularly an internal one, set a goal that relates to this? Or what about a corporate goal of ‘customer value’? How specifically should a department translate this into something concrete?

Another common experience is that the corporate strategy doesn’t cover some of the things that really matter to an individual department— such as equipment reliability for the maintenance department, employee competence for the human resources department, or supplier relationships for the purchasing department. If the goal is to clone the corporate strategy at the department level, where does an individual department make space for these types of issues in their operational strategy? Leave them out? Tack them on at the end under a catch-all result area called ‘operational issues’?

When a corporate target is set and cascaded to every department equally (i.e. every department has to achieve the same numeric level of performance), many departments are faced with making a change so large their allocated resources are completely insufficient to achieve it, or it will directly prevent them from achieving or even maintaining another performance result. They are locked by the corporate target into producing a result that will actually damage the organisation. If they achieve the corporate target they will sabotage areas of their own performance, or even the performance of other parts of the organisation.

Each of these problems come from the same underlying (and very shaky) assumption—that for the whole organisation to achieve its strategic goals or targets, each part of the organisation needs to achieve similar goals or targets. This is like assuming that to make a big elephant we need to join lots of small elephants together. Of course that’s a ludicrous idea. But for some reason, we’ve been applying it to organisational strategic direction. To make an organisation, we don’t need to join lots of smaller organisations together. We need to bring groups of people together that can each perform different and complementary functions, making the whole organisation capable of performing end-to-end processes such as developing products and services that the market requires, marketing products and services to generate customer interest, and delivering products and services to satisfy the expectations of customers. And each group within the organisation needs to focus on its specific contribution to the whole. The organisation risks an overall increase in costs if some of its parts (e.g. purchasing or maintenance) reduce costs. In fact, some parts might need to increase costs for the whole organisation to reduce costs. For example, the business improvement department might need to increase costs so it can find the most sustainable ways to remove rework and waste from the organisation’s processes. This is a classic example of the phrase ‘the whole is more than the sum of its parts’.

The organisation’s processes keep it alive, just as our processes of breathing, eating, and walking keep us alive. An organisation (or person) can change or improve only by changing or improving the processes that keep it alive. An athlete is no more going to achieve a goal of racing faster by making every cell in their body race faster than an organisation is going to achieve cost reduction by making all departments reduce costs. In fact, the athlete needs many of their cells to slow down for them to race fast—for example, brain cells so they don’t distract them from their focus, or stomach cells so they don’t waste energy on digestion or anxiety.

One of the ways we go very wrong when cascading a Balanced Scorecard throughout an organisation (without using a rigourous approach such as the Balanced Scorecard Institute’s ‘Nine Steps to Success’¹⁹) is to fall into the ‘Mini-Me’ syndrome. Everyone measures the same things that are in the corporate scorecard, but uses the data relating to their own department, division or team. This thinking presupposes that cause-effect is no more than subparts adding to a natural whole (e.g. the causes of high expenditure are the levels of expenditure in each of the organisation’s departments and branches). That’s not the case. Cause-effect thinking is broader and more powerful than that. The causes of high expenditure are usually high levels of waste, rework, and redundancy in the design and execution of business processes. Different parts of the organisation contribute differently to the overall results and overall direction, so ‘Mini-Me’ cascading vastly waters down the power of a strategy.

How do we make strategy meaningfully measurable?

Before we even bother to ask, ‘So how do we measure that?’, we’ll do well to test how measurable our strategy actually is. Where it’s not measurable, we have two choices: change it, or translate it.

Of course, the best way to make our strategy measurable is to avoid all of the previously discussed problems when we create our strategies in the first place. Then we wouldn’t need this chapter in the book at all. In truth, this is a chapter about rework—what we have to redo because it wasn’t done properly the first time. Until we can re-engineer our strategy development process to avoid producing an immeasurable strategy, one solution is to change our strategy. This means editing or rewriting the goals or objectives in our strategic plan (or operational plan, or any other plan for which we wish to have measures) until they are clear, concise, observable, results-oriented, and collectively tell a logical story of the direction we’re heading.

But changing strategy comes at a very high price. We’ve already invested a lot of time and effort in the painstaking process of analysis, dialogue, consensus and word-smithing to get the strategy we have. The idea of going back for another edit or rewrite might be more than we can take. So until we can re-engineer our strategy development process to avoid producing an immeasurable strategy, we’ll need to at least translate the goals or objectives we already have into words that spell out the observable results they imply. Then we can design or choose meaningful measures and link them back to the original objectives or goals as articulated in the plan.

If we have a strategy that isn’t measurable, here are five steps that will help translate what we have into something much easier to measure meaningfully (and consequently much easier to communicate and execute):

Step 2.1: Test the strategy’s measurability

Step 2.2: Tease out the implied performance results

Step 2.3: Map the relationships among results

Step 2.4: Test the logic of the Results Map

Step 2.5: Highlight priority results worth measuring

These steps help us draw out of our existing strategy a clear, concise and easy-to-cascade collection of performance results that can be meaningfully measured. In following these steps, we’ll create a PuMP Results Map: a single-page view of our interconnected, cascaded and measurable strategy. Let’s look in detail at each of the steps to make a strategy measurable.

The five steps to make a strategy measurable

Step 2.1: Test the strategy’s measurability

Great measures can come only from a great strategy. We need to know our goals, objectives, priorities and critical success factors—or whatever terminology our individual organisations use—before we can even have a conversation about measures. We can’t assume that our strategy is measurable. We have to test it. Here are the five Measurability Tests (A-E), that will help us to diagnose whether our goals are measurable and, if they’re not, to translate them into specific performance results.

A. IS IT A RESULT, OR AN ACTION?

If our goal is an action, then before we search for measures (and end up with trivial milestones or counts of activity) we should decide and write down the most important result we want to achieve by reaching this goal. If we attempt to measure goals that are really actions, our measures will be meaningless counts of how much activity has been done, or useless milestones that tell us nothing about what impact we’ve had.

Museum Victoria invited me to help them make the outcome areas under their strategic direction more measurable. The museum’s strategic plan project team acted as the Measures Team, and we used the five Measurability Tests on each of their goals. One of the original goals was ‘Utilise staff expertise and the unique breadth of the museum’s collection to undertake interdisciplinary research’. Is this goal really an end in itself? It sounds much more like an action than a result. The museum’s Measures Team discussed what was so important about using staff expertise and the museum’s collection. They discussed why undertaking interdisciplinary research mattered. They quickly agreed it was about collaborating to generate new knowledge about the museum’s collection, new understandings of it, and new ways to use it.

The result they arrived at was ‘Interdisciplinary and collaborative projects generate new knowledge’.

B. ARE THERE ANY WEASEL WORDS?

Highlight weasel words in the goal (or the reworded result from test A). What do those weasel words actually mean? If our goal contains weasel words, we have to rewrite it using specific language that describes observable results. Otherwise, the focus will not be strong enough to enable us to choose measures later on.

Another of Museum Victoria’s goals was ‘Develop a culture of ownership of the State collection across the organisation and within the broader community’. What exactly is a ‘culture of ownership’? The Measures Team talked about staff feeling connected to the museum and understanding the significance of the museum’s collection. They talked about staff feeling inspired and understanding their role in the stewardship of the collection. They talked about staff having a sense of pride in working for the museum. This is what they meant by this goal.

They agreed on ‘All staff know and promote the significance of the collection’. The word ‘significance’ still sounded too weasely to me, but, after some more discussion about it they explained there would be a clear understanding of what that meant within the organisation, so we left it in the result statement. If it did turn out to be too weasely to measure, Step 3 in the PuMP Blueprint (Designing Meaningful Measures) would bring the ambiguity to light, and the goal could be further reworded to be more clear at that point.

C. IS THE GOAL MULTI-FOCUS?

Are there several performance results tied up in the goal? If so, we need to write a separate result statement for each one. If we don’t, the Designing Meaningful Measures step will be so much harder. The most relevant measures come when we focus clearly on one result at a time.

Museum Victoria’s goal of ‘Develop/cultivate partnerships of national and international significance that position the museum as an integral part of the early years, primary, secondary and tertiary education sector’ failed test A because the goal was action-oriented. The revised result that the team chose was ‘Our experiences and resources are widely known and frequently used for education purposes’.

There are two specific results in this result statement that are likely to be measured quite differently. One is ‘Our experiences and resources are widely known by educational institutions’ and the other is ‘Our experiences and resources are frequently used for education purposes’.

D. SHOULD WE IMPROVE IT, CAN WE IMPROVE IT, AND WILL WE IMPROVE IT?

Is this a result that we should improve? Is it a result we can improve? Is it a result we will improve? If not, then it’s not important enough to measure and maybe not important enough to have resources allocated to it. We shouldn’t measure just because we can, or because it’s interesting. What we measure needs to be strategically important or mission critical.

Another of Museum Victoria’s original goals was ‘Construct a major collection facility that supports the guaranteed development, preservation and use of the State Collection for future generations’. As we can see, this original goal was action-oriented. After the team discussed the goal’s intent (taking proper care of the museum’s collection), the goal became ‘The unique scientific and heritage record of the State Collection is developed and preserved for future generations.’ We might also agree that this new result could be worded even more ‘measurably’ by clarifying what ‘developed and preserved’ means. Translating weasely goals into measurable results really does require practice, and we rarely master it in our first few attempts. But that doesn’t mean we ought to dwell too long in Step 2 of the PuMP Blueprint in pursuit of perfection. We’ll do our best, and then trust that Step 3 will help us fix any remaining ambiguity in our result statements.

The Measures Team definitely believed this was a result that they should improve, and was therefore important to measure. However, it’s only a result they can achieve if they get access to much-needed funding to build the new collection facility. So either it will become a result they won’t be measuring until they get the needed funding, or it will be a result they will go ahead and measure anyway. But they won’t set unachievable targets for how much of the collection they can develop or how well they can preserve it in the absence of the new facility.

E. HOW DOES THE GOAL ALIGN WITH THE REST OF THE STRATEGY?

As we see measurable performance results emerging from our original goals, we can more easily see how the goals relate to one another. This is useful, because it helps us link the results into a complete story of our strategy. This won’t be just a list of goals; it will become (in Step 2.3) a Results Map. For now though, the question is: to which other results within our organisational or business strategy does this result have a cause-effect relationship?

Recall one of Museum Victoria’s results: ‘Interdisciplinary and collaborative projects generate new knowledge’. They quickly identified that this result had a strong cause-effect relationship with another result: ‘The museum’s experiences and resources are widely known and frequently used for education purposes’.

Don’t ask the question ‘how will we measure this?’ yet. It’s important not to get caught up in how hard it might be to implement likely measures at this stage. We will address this concern rigorously in Step 3 of the PuMP Blueprint, where we often discover that it’s easier to measure our results that we first assumed. Working on one specific step at a time, and not racing ahead prematurely, is a great strength of the PuMP Blueprint.

Step 2.2: Tease out the implied performance results

Analysing goals by using Measurability Tests A through E in Step 2.1 entails digging deep into the real meaning of our goals. We need to get to the point of clearly articulating the observable or detectable results implied by the goals—the differences we are trying to create through our business, organisation, department or team. I call this ‘teasing out the results’ from our strategy because we very likely already know and have talked about the results that truly matter. They simply didn’t make it into the words we ultimately chose to articulate our goals in the planning document. Wouldn’t it be great if we used rich, evocative language in our plans to clearly describe the results we want to achieve?

But there’s one very important thing to bear in mind: performance results are a clear and specific expression of the intention of the goal, and not merely an attempt to reword the goal better. Two different organisations might have an almost identically worded goal, but could end up with very different performance results. The process of teasing out the implied performance results strips away the words of the goal until we arrive at what the words really mean and what we really want.

We’ve been using the process that took place at Museum Victoria to illustrate the steps in making goals measurable. Now we are going to shift to a ‘composite’ museum example based on the experiences of several different museums. (We’ll call it ‘World Museum’.) Table 2 shows a complete set of original goals for World Museum, and the new performance results that emerged from applying Step 2.1 in the process of making the goals measurable. You will recognise some of Museum Victoria’s goals that we have already discussed as well as some additional ones.

Table 2 illustrates the transformation from immeasurable goals to performance results that are simple, easy to understand, and more measurable than the originals. It’s going to be easy for you to critique these performance results even further, and see ways they could be more measurable, more meaningful, and more understandable. But remember that the idea is not to polish the goals to perfection at this point. We are focusing on improving their measurability to the point where the Measures Team feels the goals are ‘almost there’. The next PuMP Blueprint step (Designing Meaningful Measures) will take care of any remaining ambiguity.

TABLE 2. Teasing out performance results from World Museum’s goals

At this point we have a choice. We can rewrite our plan with the new, measurable, descriptively articulated performance results, or just use this work to help us get to meaningful measures. Either way, a useful next step is to bring these performance results together in a visual story of our strategy as a whole. This will be a great aid in ultimately cascading that strategy throughout the organisation or business.

Step 2.3: Map the relationships among results

Great measures will collectively tell the story of how the whole is performing—the whole organisation, the whole department, the whole team. To understand the performance of the whole, we need to understand how the most important results relate to one another. It’s important to consider more than just cause-effect relationships. We need to consider conflict and companion relationships too. The cause-effect relationship is the easiest to grasp—by achieving Result B, we will more likely also achieve Result A. For example, achieving the result of high customer loyalty will mean we are more likely to achieve the result of greater profitability.

But cause and effect is not the only kind of relationship we need to monitor when we manage organisational performance. We want to be sure that, as well as achieving the result of high customer loyalty, we are also achieving the result of healthy profit margins on sales to existing customers. We won’t get greater profitability unless our loyal customers are also profitable to us. So there is a companion relationship between the result of high customer loyalty and the result of healthy profit margins on sales to existing customers. When we measure companion results like these, we track the measures together for a complete picture.

The conflict relationship applies when two results aren’t compatible. For example, we might have another performance result in our strategy that has to do with customer account managers increasing the time they spend with existing clients to deepen rapport. But this is in conflict with another result (‘to attract new customers at a higher rate’) because achieving these two results competes for the customer account managers’ limited time.

Understanding these three relationship types helps us to meaningfully cascade the corporate strategy throughout the organisation. Instead of cascading strategy by getting every department to adopt or adapt a duplicate of the corporate strategy, we change the questions we ask and engage each department to consider the impact they have on the corporate strategy. Instead of the typical questions (‘What should our department’s customer focus goal be?’ or ‘What should our department’s cost reduction goal be?’), we need to ask questions like ‘In what ways does our department impact corporate customer focus?’ and ‘In what ways does our department impact organisational costs?’ The answers to this second set of questions are often totally different from the answers to the first.

For example, instead of choosing a departmental goal of internal customer satisfaction because the corporate goal is about customer satisfaction, the department could end up with goals for:

service delivery cycle time

product reliability

billing accuracy

consistent pricing

fast responses to customer enquiries

providing technical solutions in lay terms for the sales team to use in responding to customer complaints.

The department would decide on results to do with their process, and examine how well this process currently supports the corporate priorities. Cascading strategy this way is about understanding the unique impact each area or process has on improving the organisation’s capability to achieve its strategic vision. Don’t you agree this is more sensible than the Mini-Me approach?

We create a Results Map to illustrate relationships among our results. A Results Map is constructed by placing all of the newly articulated performance results into bubbles, and then linking them together using the information written down in response to the Measurability Test E from Step 2.1 which asks how each result aligns with the rest of the strategy. Figure 4 shows an example of a Results Map prepared by a fire department.

Before we go any further, let’s use the example Results Map in Figure 4 to define the three specific elements that make up the anatomy of a Results Map.

THE ANATOMY OF A RESULTS MAP

Let’s look at the three elements of a results map: the layers, the bubbles, and the lines connecting the bubbles.

Layers The first part of a Results Map’s anatomy is the layers. There are four layers, denoted by colours. In the Results Map in Figure 4, we can see an orange layer around the outside, a blue layer inside that, a green layer inside that, and finally a pink layer in the centre.

And each layer contains a specific category of information.

The pink layer is the collection of results that describe the mission and vision (purpose) of the organisation. These are the ultimate results the organisation or business intends to achieve, and they usually change very slowly (if at all) over time. These results might have a 10- to 20-year timeframe or even longer.

The green layer contains the results implied by the current strategic goals for the organisation or business as a whole. These goals are about what needs to improve overall for the organisation to better fulfil its mission and move closer to the vision. These goals will have a two- to five-year timeframe. They will change as the organisation grows its capability and new improvement priorities emerge.

The blue layer contains the most important goals for business processes, such as the maintenance process, customer delivery process, marketing process or procurement process. If an organisation is not process-oriented, then the blue layer will contain the most important goals for the organisation’s main functions, such as human resources, operations or marketing. Either way, the blue results are about what needs to improve for the organisation to achieve the green results. Results in the blue layer will have a one- to two-year timeframe.

The orange layer contains results that are specific to parts of business processes or business units within the main functions of the organisation. These results are about what needs to improve at a specific operational level to achieve the blue results. Results in the orange layer can have a timeframe anywhere from a few months to a year or so.

FIGURE 4. Results Map for a fire department
FIGURE 4. Results Map for a fire department

In the early days, when I was inventing PuMP, it turned out that every time I facilitated a Results Map for people it worked out to have four layers. I have seen some organisations add a fifth layer, but I haven’t seen anyone reduce the layers to fewer than four. If there is a fifth layer, it is usually in the maps of large multi-national corporations that, because of their breadth and size, have a more complex organisational structure than smaller organisations.

Bubbles The second part of a Results Map’s anatomy is made up of the ‘bubbles’ that are colour-matched to, and positioned within, each of the four layers. Each bubble contains a specific performance result, derived from the five Measurability Tests in Step 2.1.

Lines The third part of a Results Map’s anatomy comprises the lines that connect the bubbles to one another. These lines are customized to represent the relationship types I mentioned earlier: cause-effect, companion, or conflict.

The cause-effect relationship is depicted by a line with an arrow, starting at the cause and pointing to the effect. In the fire department’s Results Map in Figure 4, the green result of ‘fires are prevented’ has a cause-effect relationship to the pink result of ‘less injury from fire incidents’.

The companion relationship is depicted by a line with a solid dot on both ends that I like to think of as holding the two results together. There is a companion relationship between ‘enough fire fighters are available for the demand’ and ‘equipment is available when needed’ in the fire department’s Results Map in Figure 4.

The conflict relationship is depicted by a line with an arrow on both ends, but a different style of arrow from that used for the cause-effect relationship. In the fire department’s Results Map there is a conflict relationship between ‘crews arrive at emergencies quickly’ and ‘response doesn’t put the community at risk’.

One of the benefits of mapping the relationships among results is we can quickly check which results are truly strategic, and which might belong instead in a department or business unit plan. In Figure 5 we can see a cause-effect relationship between the two performance results derived from World Museum’s original goal of ‘Develop/cultivate partnerships of national and international significance that position the museum as an integral part of the early years, primary, secondary, and tertiary education sector’. The performance result ‘The museum’s experiences and resources are widely known by educational institutions’ is a cause of the result ‘The museum’s experiences and resources are extensively used in education’. In other words, one way to increase the museum’s use in education is to increase educational institutions’ awareness of the museum’s experiences and resources. It might then make sense to move ‘The museum’s experiences and resources are widely known by educational institutions’ to the blue layer (the layer for business or organisational processes) in the Results Map.

FIGURE 5. Results Map for World Museum
FIGURE 5. Results Map for World Museum

After the corporate strategy is mapped (as it has been in Figure 5 for our museum example) the Results Map becomes the foundation for teams within the organisation to identify their own goals in contributing to the direction represented in the map. They identify their business processes or workflows, define the results that will have the most effect on the strategic results, and analyse how they need to change those processes to help achieve the strategic results. That’s cascading strategy.

IS A PUMP RESULTS MAP THE SAME AS A BALANCED SCORECARD STRATEGY MAP?

The PuMP Results Map and the Balanced Scorecard Strategy Map are both visual tools that highlight the cause-effect relationships among the elements of an organisation’s strategy. But there are some key differences.

The PuMP Results Map uses results-oriented language to make it easy to understand the specific change we’re trying to create. In contrast, a Balanced Scorecard Strategy Map tends to use high-level action language. So in a Balanced Scorecard Strategy Map, we might see a statement like ‘Train the workforce’ whereas in a Results Map we would see something like ‘Employees know how to put the customer first’. The Results Map statement in this case describes specifically what we are training the workforce to be more capable of.

Part of the discipline of creating a PuMP Results Map is expressing each result in unambiguous, measurable language. Most Balanced Scorecard Strategy Maps contain brief descriptions that are often immeasurable because of weasel words and ambiguity. For example, a Strategy Map is likely to contain a phrase like ‘Optimise staff efficiency’ whereas a Results Map might express this using two results that have a conflict relationship: ‘Staff produce their widgets quickly’ and ‘Widgets are produced without error’.

Another difference is that Strategy Maps contain only cause-and-effect relationships, whereas PuMP Results Maps include companion and conflict relationships. Including all three types of relationships is important in interpreting and responding to performance results to maximise performance of the whole organisation instead of the parts, which can often sabotage performance of the whole.

The PuMP Results Map is not tied to a specific framework, whereas the Strategy Map is structured around variations of the Balanced Scorecard’s four perspectives. Although we can bring the four Balanced Scorecard perspectives into a Results Map, we don’t have to.

Another helpful feature of the PuMP Results Map is that it visually highlights how a strategy cascades throughout an organisation, using a radial layout to easily fit several layers of strategy on one map. This layout helps cascade strategy because teams throughout the organisation can see a clear line of sight to the corporate goals as they focus on the aspects of corporate strategy on which their processes have the most impact.

One of my customers, Anne, is the Business Strategy Team Manager in a UK government agency. Anne uses Results Mapping with her colleagues, and has had some great results. For example, two teams that were brought together under one manager worked alongside each other for a couple of years feeling they both had very different roles and issues. Results Mapping helped integrate them by showing they were both focused on delivering the same results through their different activities. Their Results Map helped them see lots of synergies between what each team did, and the individual team members immediately started identifying opportunities for jointly delivering the desired results. Six months later, engagement in the now-unified team had greatly increased, with team members stating they felt like a single integrated team and were much more focused on achieving their goals rather than carrying out traditional activities.

By contrast, it usually takes several Strategy Maps at different tiers of a Balanced Scorecard implementation before frontline staff can see how their goals link through to the strategic goals. But all layers or tiers of an organisation’s strategy can be displayed on a single PuMP Results Map, so Results Map makes it easier to test how sensible a story the entire strategy tells.

Mapping the relationships among our results is a first step in seeing the whole strategy in a single view. It quickly makes apparent any gaps in thinking, any duplication, and any results that don’t have an obvious link to the strategic direction.

Step 2.4: Test the logic of the Results Map

Our Results Map shows the clear, measurable results implied by our strategy and lets us see both the big picture and the details at once, which opens the door to lots of insights. Identifying the relationships that link results to one another makes it easy to see two important things:

how teams and business units align with the corporate strategic direction

how sensible a story the results and their relationships tell about how the strategic direction is to be achieved.

HOW WELL DO TEAMS AND BUSINESS UNITS OR BUSINESS PROCESSES ALIGN WITH THE CORPORATE STRATEGIC DIRECTION?

Strategic directions are often written in so much detail that they end up containing many operational results—results that should be owned and measured by teams or business units within the organisation. An organisation’s performance results don’t exist on a single logical level, but rather in hierarchies of cause-effect. That’s what strategic alignment means: every team or work group throughout the organisation or business is focused on the most important results for which they are responsible, and achieving those results will directly help to achieve the corporate results. For example, the billing team focuses on invoice accuracy and on-time payment of invoices, which contributes directly to the strategic result of healthy cash flow.

When you use a Results Map to visually lay out the results from Step 2.2 and 2.3 of the PuMP Blueprint, you might notice that some results have a chain of cause-effect relationships to one another. Often the results at the ‘cause’ end of the chain aren’t actually strategic results, even though they were teased out of the strategic goals. Instead, they might be operational results belonging to a business unit or team, and therefore more appropriately situated in the blue or orange layer of the Results Map. These results are likely to be very important for the business unit or team to measure and monitor as part of their contribution to the strategic direction. But they are no less important than the green or pink strategic results. Things in an outer layer of the Results Map do not necessarily have lower priority than things in other layers. Everything in a Results Map is important because this entire collection of results is essential to achieving the strategic direction.

We saw an example of this in World Museum’s Results Map. A cause-effect chain of three results appeared, from the blue result of ‘The museum’s experiences and resources are widely known by educational institutions’ through to the pink result of ‘People feel inspired to better care for their natural and cultural world’, as shown in Figure 6. It’s likely that the Marketing department would own the blue result because this result has everything to do with how the museum positions its brand and promotes its products and services.

When the results are placed in their appropriate layers in the Results Map, we can see the story of the strategy begin to emerge.

FIGURE 6. A cause-effect chain of results from World Museum’s strategic goals
FIGURE 6. A cause-effect chain of results from World Museum’s strategic goals

HOW SENSIBLE A STORY DO THE RESULTS AND THEIR RELATIONSHIPS TELL ABOUT HOW THE STRATEGIC DIRECTION IS TO BE ACHIEVED?

The relationships among results—cause-effect, companion, and conflict—are really hypotheses about the drivers and levers that will maximise our chances of achieving the corporate goals. Some of these hypotheses might have already been tested by past experience or experiment, such as the hypothesis that increasing customer loyalty will increase profits. But some of the hypotheses depicted in the Results Map might not yet be tested, such as the hypothesis that customer account managers increasing the time they spend with existing clients will cause an increase in customer loyalty.

Step 2.4 can reveal flaws in the logic that underlies a strategy. When we have our strategy laid out visually in a Results Map, a brilliant way to test its logic is to describe what the map is saying in everyday language, as though we’re explaining it to someone who hasn’t seen it before or hasn’t been exposed to our organisation’s strategy. We put our Results Maps into our own words, telling the story out loud to check whether it makes sense.

When we do this, do any shaky assumptions suddenly become obvious? Are there any wide-open gaps where strategic results have no cause-effect relationships with tactical or operational results? Are any results still too vague or ambiguous (weasely)? Are results positioned in the best place or layer in the Results Map to make the cause-effect flows sensible, make the story of the strategy easy to understand, and hold it together logically?

In his book Communicating Strategy,²⁰ Phil Jones describes an example of how each member of an executive team tells the story of their strategy:

The chief executive and her directors presented their seven strategy maps. Each took roughly 5 to 10 minutes to tell the story from their map, before moving on to the next director. Within an hour, the executive team had told the whole story of their strategy. For the next 2 hours, the middle managers were let loose on A0-sized versions of the strategy maps and were invited to comment on them, ask questions and add their thoughts about where they could help to deliver the changes.

The energy in the room was extremely high and the conversations, involving people from widely differing departments, were animated and enthusiastic. The chief executive described the event as the single most effective thing that was done to communicate the strategy and engage the middle managers’ enthusiasm and ownership of the strategy.

What Phil Jones describes here is very similar to another PuMP technique called the Measure Gallery. (We’ll read more about the Measure Gallery in Chapter 7, which describes Step 4 in the PuMP Blueprint.) But before we start to communicate our strategy widely (and certainly before we put effort into measuring it) we ask the team that created the strategy to test its logic and take the time to:

remove duplication

clarify ambiguity

correct links in relationships

fill important gaps

locate results at the appropriate place in the natural hierarchy of strategic, tactical and operational levels.

Step 2.5: Highlight priority results worth measuring

Measurement takes effort, and often our goals, objectives and strategies imply so many results that measuring them all is too overwhelming. So we don’t always measure every result implied by our strategy. Instead, we highlight the results that are most important to measure now. In time, we might measure more, fewer or different results as we learn more about what really matters most.

Of course, the ideal is that every result implied by our strategy—or every result we’ve mapped into our Results Map—should be measured. If it matters enough to pursue and commit resources to, then it matters enough to measure. There’s a big risk in putting resources into an initiative to achieve a result when we have no way to measure whether the result is indeed being achieved.

But let’s be realistic. In organisations where strategy design is not yet an instinctive competence—which is probably most organisations —people often end up with too many results to achieve, too many goals, too many objectives, or too many priorities. Measuring them all is both overwhelming and unnecessary, and it’s likely that time and money will run out before all of these goals can be achieved. No matter how convincingly we try to explain this, people usually won’t reduce the number of results they aim for until they have repeated personal experiences of the undesirable consequences of over-promising and under-delivering. So in this instance, where there’s simply too much to start measuring all at once, Step 2.5 is a blessing. We highlight the results in the Results Map that truly are the most important to start measuring now, and we give ourselves permission to worry about the rest later.

For our World Museum example, Figure 7 uses thick-bordered bubbles to show the highest-priority results that should be measured first.

When the Measures Team gets their Results Map to this stage, we need to prevent them from dwelling too long on it. I’ve noticed a tendency for people to want to perfect the Results Map before they move on to designing measures. But the Designing Measures process does the best job of improving the Results Map. So we need to make sure that our PuMP Pilot Measures Team is ready to start designing powerful performance measures when we meet for Step 3 in the PuMP Blueprint.

Take Action: Create Your Results Map

You won’t fully appreciate the full power of a Results Map until you’ve spent an hour or two creating one. The power is in the dialogue, and the map itself is a visual way of capturing that dialogue and drawing out the things to focus on. Make sure your results are written in measurable language—no weasel words allowed.

Go to the book’s website (www.pumpblueprintbook.com) and download the PuMP Results Map template.

FIGURE 7. The priority results for World Museum to measure first
FIGURE 7. The priority results for World Museum to measure first

Chapter summary

A good strategy—one that is easy to communicate, understand and measure—has certain hallmarks. It is articulated using clear, unambiguous, plain-English words. It describes results that matter before it discusses actions. It is specific and focused, not a collection of different results linked together with commas into a single sentence and masquerading as a single goal. It is cascaded by exploring how each part of the organisation uniquely impacts the achievement of corporate goals.

We need to pass our goals through the five Measurability Tests to make sure they truly are clear and specific enough to measure in a meaningful way. Most goals are not written in measurable language, which is the biggest reason that finding good measures seems so hard.

The PuMP Results Map is a visual tool that helps people focus simultaneously on both the big picture of the corporate strategy and the details of their own contribution to it. The process of creating a Results Map encourages us to be results-oriented, specific, clear in our language, rigorous with our logic regarding how results relate to one another, and ruthless about priorities. The Results Map is a tool that makes communicating and cascading strategy easy.

Now that your Measures Team is clearer about the performance results that matter, it’s time to find meaningful measures for those results.

6

Step 3: Designing Meaningful Measures

‘My task, which I am trying to achieve is, by the power of the written word, to make you hear, to make you feel—it is, before all, to make you see.’

— Joseph Conrad, Lord Jim

I love a sunburnt country,
A land of sweeping plains,
Of ragged mountain ranges,
Of droughts and flooding rains.
I love her far horizons,
I love her jewel-sea,
Her beauty and her terror -
The wide brown land for me!

— Dorothea Mackellar, from My Country

Most of us aren’t conscious of the approach we use to find or choose performance measures. Nor are we aware that our words lay the path to the most powerful performance measures. Which is why we are also usually unhappy with the selection of performance measures we have.

Selecting performance measures is usually treated as a trivial, pesky task imposed on us by an annual strategic or business planning workshop. Many of us take the fastest route to finalising a list of performance measures in the KPI column of our plans and, more often than not, the resulting KPI column contains a list of things that are simply not good performance measures, if they are measures at all. (As we’ve discovered in previous chapters, many people don’t know what a real performance measure is.)

We don’t have to look too far to find examples of meaningless performance measures. A quick Internet search for ‘strategic plan’ will return plentyof them. (Sadly, most of the strategic plans won’t have any measures.) When I recently did this Internet search, the first five organisations’ strategic plans listed had no measures whatsoever. The sixth listing was for Rowing Australia, the national governing body for the sport of rowing, whose vision is for Australia to be the world’s leading rowing nation.

One of Rowing Australia’s strategic priorities is ‘High Performance’, and their objectives under this priority include having the most successful international program in world rowing and winning three gold medals at the 2012 London Olympic Games. One of the measures for this priority was ‘National High Performance Plan in place and national requirements adopted’. As you might recognise after reading the first few chapters of this book, that’s not a performance measure at all. It’s a milestone in the process of pursuing high performance, but it’s not evidence of high performance. Either the plan is in place and the requirements are adopted or it’s not. Milestones are for program management. They are not performance measures.

Another measure for this priority was ‘100% engagement by states in the National High Performance Plan’. What do you suppose they meant by ‘engagement’? Is it unanimous acceptance of the plan? Is it everyone’s involvement in the creation of the plan? Is it everyone’s enthusiastic implementation of the plan, or pursuit of the plan’s goals? And who specifically in the states ought to be engaged? The vagueness of this measure makes it impossible to determine whether Rowing Australia has carefully chosen and defined this measure, or whether it’s just an idea they had and hoped to flesh out in quantitative detail later.

This next plan in my Internet search results that included performance measures was the tenth plan listed: the Queensland Department of Premier and Cabinet’s 2009–2013 Strategic Plan. One of their objectives is ‘Supporting the Premier by maximising opportunities for Queensland’, and their measure for this objective is ‘Director-General satisfaction with the quality of support provided for the Council of Australian Governments (COAG), Council for the Australian Federation (CAF) and senior officer meetings’. Does the Director-General’s satisfaction provide any meaningful evidence of maximising opportunities for Queensland? No doubt it’s easy to measure—just ask the Director-General how satisfied he feels with the quality of support provided. But easy to measure doesn’t mean meaningful to measure.

Another objective from this Department’s 2009–2013 Strategic Plan is ‘Building a strong arts sector that enriches the lives of all Queenslanders’, measured by ‘Level of attendance and participation in subsidised arts sector events’, ‘Rates of volunteering in arts and culture’, and ‘Income and employment opportunities generated by the Cairns Indigenous Art Fair’. Perhaps the first two measures are evidence of the strength of the arts sector. But if this objective is honestly about all Queenslanders, how could the third measure even come close to gauging how well the lives of all Queenslanders are being enriched by the arts?

The next strategic plan I found with measures (the 11th title that Google returned) was the 2009 Strategic Plan for the University of Western Australia’s Human Resources (HR) department. Their plan consisted of 11 goals and 26 objectives organised around six Strategic Directions²¹. It also included 58 performance indicators with between zero and six for each objective. That is a lot of performance measures. But when we take a closer look, it’s easy to see how that happened. For the HR department’s objective ‘To provide appropriate leadership development opportunities’, they chose four performance indicators:

Increased participation rates.

Positive participant satisfaction surveys.

Positive changes in leadership behaviours and cultural indicators as measured through the WLS and other tools.

Increase in the proportion of staff indicating on the Working Life Survey their satisfaction with the quality of local management.

They’ve done a very good job at focusing their performance indicators on the results they want from that objective, which is likely to be that managers throughout the university are better leaders. But their objective is worded in an action-oriented rather than results-oriented way. By writing this objective using action-oriented rather than results-oriented language, they have listed a range of performance measures that relate to the leadership development process. Although the last two performance indicators are evidence of the outcome of that process (i.e. that managers become better leaders) the first two are simply evidence of participation in the leadership opportunities, which are drivers of the outcome.

If the HR department had a Results Map (PuMP Blueprint Step 2), it might look like Figure 8.

In this HR department’s Results Map, we might see a blue result reading ‘managers throughout the university are better leaders’. Then, at the orange level in the Results Map, we might see two results that have a cause-effect link to the blue result: ‘managers readily participate in leadership opportunities’ and ‘managers find the leadership opportunities valuable’. These two orange results are measured by the first two performance indicators for the HR department’s first objective. When outcome and driver measures are included in a strategic plan, the result is a lot of performance measures. Driver performance measures actually belong to teams or business units because these measures assess the results of work at that level in the organisation’s structure. When this isn’t clear people can get confused about how to cascade a strategy, and attempt to develop even more detailed objectives and performance measures for their own team or business unit. And these measures are often trivial counts of workloads or project milestones.

It’s also worth pointing out that the first three performance indicators lack specificity. We’d probably assume that ‘Increased participation rates’ would be measured by the percentage of managers throughout the university who participated in available leadership development opportunities. We might also assume that ‘Positive participant satisfaction surveys’ would be measured by participants’ average overall satisfaction rating of their experience in a leadership opportunity. But the performance indicator of ‘Positive changes in leadership behaviours and cultural indicators as measured through the WLS and other tools’ alludes to some measures about behaviour and culture without specifying them. It’s problematic when performance measures are not clearly detailed in a plan because ambiguity delays their reporting, and time is wasted deciding exactly which measures will be used when the plan has failed to make that obvious.

Figure 8. An excerpt of a possible Results Map for the HR Department
FIGURE 8. An excerpt of a possible Results Map for the HR Department

Listing 13 in my Google search was the Strategic Plan for the NATO Consultation, Command and Control Agency (NC3A). This plan used a Balanced Scorecard approach, and had a total of 25 performance measures for 16 objectives. Their first objective was ‘Be relied upon as a trusted Alliance Partner’, and their performance measure for this was ‘NC3A contributions to NATO’s success’. What do you suppose they meant by ‘contributions’? This is not a measure at all because we can’t tell exactly what is being counted or quantified as evidence of the degree NC3A are being relied upon or trusted. About three-quarters of the ‘performance measures’ in this plan are similarly vague statements that don’t make clear the evidence or quantification of the objectives. For example, for the objective of ‘Deliver coherence to NATO C4ISR’, a performance measure was ‘Alliance and Nation’s coherence’. For the objective of ‘Develop and grow relationships with external stakeholders’, the performance measure was ‘Key Stakeholder Relationships’. For the objective of ‘Implement top performing CATs [as teams]’, the performance measures were ‘Performance Management’, ‘Management of Staff (Internal)’, and ‘Management of Staff (External)’. The wording of these ‘measures’ is so broad and vague there is no way to tell what kind of data should be gathered to measure them, or how it should be analysed to obtain evidence of whether these objectives are being achieved or improved as time goes by.

The ACT Department of Education and Training Strategic Plan was the fourteenth result that came up in my Internet search. Their plan was simple and focused, and for the most part their performance measures were clear and well-aligned with their priorities. Their first goal was ‘To ensure students succeed through quality teaching that engages them and supports the development of capabilities for life’. This goal had three priorities:

Improve the quality of teaching through relevant professional learning, high-quality curriculum and pedagogical support.

Improve literacy and numeracy outcomes for all students.

Close the learning achievement gap between Aboriginal and Torres Strait Islander students and other students.

For these priorities were three performance measures:

Proportion of students in all achievement bands above the national minimum standard in the National Assessment Program—Literacy and Numeracy (NAPLAN).

Year 12 mean scores and completion rates.

Teacher retention and satisfaction.

We can see the alignment quite clearly between the first two measures and the second priority. But there are no specific measures that align with the first or third priority. And the third measure doesn’t really provide direct evidence of any of the priorities. Looking again at the goal, we should ask. Do any of these measures provide evidence of the degree to which students have developed capabilities for life?

We’re up to the sixteenth result that came up in my search—Access Arts Inc., a not-for-profit organisation that helps Queenslanders who have disabilities to participate in the arts. And their plan included performance measures. One of their objectives was ‘Developing the capacity for research and evaluation underpins focused activity and ensures a relevant service’, and was measured by ‘Effective research and evaluation procedures using both formal and informal methodologies are in place by mid 2012’. Another objective was ‘Understanding our purpose, target markets and the marketplace creates a strong brand and meaningful communications to reach out to new and existing stakeholders’, and was measured by ‘By 2013, the Access Arts “story” and suite of services will be easily understood and consistently and appropriately communicated to all stakeholders through a variety of media’. Like one of the examples we saw earlier in the Rowing Australia strategic plan, these are milestones, not performance measures. Having procedures in place by a particular date is not evidence that capacity for research or relevance of services have increased. Milestones describe the date by which we want a particular initiative or project or activity to be completed. And we already know that completing an activity does not mean a performance result or outcome has been achieved.

Let’s look at the big picture of what my Google search returned. Of the first 20 listings, 12 of the organisations had no measures in their strategic plans, and the remaining eight included measures that were too vague, not relevant to the goals, or not really measures at all. Granted, the organisations whose strategic plans Google will find are government or not-for-profit organisations because most privately owned companies do not publish their strategic plans publicly. (They want to protect their competitive positions in their markets, and there is no practical or legislative need to do so.) But do you think the measure selection situation is any different in the private sector? From years of experience, I’ve seen that some business results in the private sector (e.g. profit, revenue and sales) are easier to measure than the types of results that public-sector and non-profit organisations seek. But I’ve seen the private sector struggle just as vainly to meaningfully measure other business results such as customer relationships, employee engagement, process productivity, and product and service quality. Google also returned mostly Australian organisations in my search (presumably because I am based in Australia). But do you think the rest of the world is any better at performance measurement than the organisations I reviewed? From years of working with organisations and companies around the world, I’ve seen the same amount of struggle to find meaningful measures.

At the root of the poor performance measures so many organisations are trying to use is the approach people take to find the performance measures in the first place. Let’s take a look at the approaches that are typically used, and why they don’t work.

Traditional approaches to selecting measures, and why these approaches fail

The most common method organisations use to select performance measures is a combination of the following:

Brainstorming measures, where participants list as many potential measures as they can think of and then do some kind of short-listing.

Looking to existing data or measures to save the costs of measuring something new and having to collect more data.

Adopting measures from industry benchmarking, or some other version of adopting measures from other organisations in similar sectors or industries.

Measuring what stakeholders (e.g. the board, regulators or industry bodies) want to measure.

Outsourcing the selection of measures to experts, consultants or industry specialists and accepting what these experts ‘know’ are the right measures.

Each of these methods certainly has strengths. But we often forget to examine the drawbacks. And the drawbacks are the reason why most organisations have too many of the wrong measures, and not enough of the right ones.

Brainstorming performance measures

The first approach, brainstorming, usually entails looking at an objective or goal and asking a simple question: ‘What measures could we use?’ Everyone sits around a table and randomly throws out potential performance measures for that particular objective or goal as related to the function or process that their departments oversee. This process might produce a list that looks something like this:

Measures for Staff Engagement:

Turnover

Sick days

Retention rate

Introduction of talent management

Overtime

Staff survey

Engagement index

Staff satisfaction with their job

Leadership development

Performance management

Brainstorming has many positive attributes: it can generate lots of ideas for measures quite rapidly, it’s easy to do, it does not require special knowledge or skill, it’s familiar (so it won’t require distracting explanations or new steps), and it can be very collaborative and engage people to be part of the process of selecting measures. But brainstorming rarely produces good measures.

The truth is, we’re not really finished after the brainstorming is over. We still have to work out how to get a final selection of measures from that long list. Voting or ranking the ideas and skimming the few that rise to the top is not the answer. Ideas for potential measures need to be vetted or tested to weed out the items on the list that are not measures at all. Looking at the example list above, we can see that ‘Introduce talent management’, ‘Leadership development’ and ‘Performance management’ are actually actions or initiatives. They might improve staff engagement, but they are not measures because they don’t provide evidence of the actual level of staff engagement. ‘Staff Survey’ is a means to collect data, but it doesn’t elaborate on what kind of data for the computation of which specific measures. These four suggestions simply don’t belong on a list of measures.

Often the people involved in brainstorming measures are brainstorming measures for different goals without realising it. They start with fractured understandings and presumptions about what the goal means. The process of brainstorming does not offer an opportunity for testing whether the goal means the same to everyone. In the example brainstorm of measures above it’s very likely that ‘Turnover’ and ‘Retention rate’ were suggested by those who thought staff engagement was about people wanting to stay with the organisation. And the measure ’Staff satisfaction with their job’ was probably suggested by those who understood staff engagement as people being interested and motivated in their work.

Some of the measures produced in the example brainstorm session have little to do with what staff engagement looks like when it’s happening. ‘Retention Rate’ and ‘Turnover’ are well-known measures used by Human Resource departments, but are they really good evidence of staff engagement? You might say yes, because if staff are less engaged, the organisation will have a hard time retaining them. But these two measures give direct evidence only of how well an organisation holds its staff base stable. They are not direct evidence of staff engagement. All we need to do to prove it is observe that staff engagement can indeed be quite low, with staff feeling little passion or commitment to the goals of the organisation, while the ‘Turnover’ measure shows a very low percentage of staff being lost from month to month. Even though people might feel uninspired, that doesn’t mean they quit. There are many reasons people keep jobs they don’t like or don’t care about. A culture where employees feel disempowered, apathetic, and exhibit many traits of learned helplessness is one example where lack of engagement and a low turnover rate can coexist. In such a situation, decision makers would be incorrectly interpreting a low ‘Turnover’ as strong staff engagement.

Many brainstormed measures are too vague for us to even get an operational handle on what the measure is. After the brainstorming session, and someone is tasked with implementing the chosen measures weeks later they might have little idea of how to calculate them or find the right data for them. For example, ‘Engagement Index’ is a vague concept that gives no clues whatsoever about how to construct the measure. If it is already a known and defined measure, then that needs to be made clear by a reference to its definition or source in the strategic or business plan whose performance measure column it occupies. There are too many examples of vague measures that never get implemented or reported because it’s anybody’s guess as to what in the world they mean.

And finally, brainstorming does not challenge our thinking beyond measures of which we are already aware. For example, when the above list of potential measures for staff engagement was brainstormed, no-one suggested Gallup’s Employee Engagement Ratio.²² It’s a performance measure used by organisations such as Google and FedEx, who are renowned for having highly engaged staff.

An approach to selecting meaningful performance measures needs to produce a few measures that have been thoroughly tested for their relevance or strength in the goal or result for which they are selected. Brainstorming doesn’t reliably achieve this. Brainstorming was created by Alex Osborn early last century as a creativity tool.²³ It is not a performance measure design tool. So save brainstorming for coming up with the theme for the company picnic.

Looking to existing data or legacy measures

Another common approach for selecting measures that people don’t seem to critique (but should) is to go looking for performance measure ideas in their existing data (i.e. asking the question ‘What can we measure from the data we’ve already got?’). Or they select from a list of legacy measures they’ve used for years already. Someone will say, ‘We need a measure for customer service performance .  .  .’ and someone else will respond with, ‘Well, we already have data on customer complaints, so let’s use Number of Customer Complaints’.

Obviously this approach to selecting performance measures is easy, quick, and low cost. People likely share a common understanding of the existing measures already, and there is a treasure trove of historic data available for trend analysis. These are all very good reasons why this is such a comfortable, common approach. But they are not good enough reasons to justify using this approach.

The predominant flaw with relying on legacy measures is it is symptomatic of an organisation not open to challenging whether the data they are collecting is really capable of telling them what they need to know about where they are going. If we measure what we have always measured, we will get what we have always gotten. What does the number of customer complaints really tell us? Is it really directly correlated with the number of customers who aren’t happy with service performance? No. It’s correlated with two things: the number of customers who feel inclined to complain, and the willingness of employees to capture the complaints that are made. The best way to measure customer service performance from the customers’ perspectives is to collect data about their perceptions.

Then there’s a flipside of not collecting the data an organisation really needs: the waste involved in collecting data that no longer supports the direction and focus the organisation has today. Measuring something just because it’s always been measured, and collecting data just because it’s always been collected, distracts people from the organisation’s strategy. And in many cases pits them against it. An example we’ve discussed before is the common practice for call centres to measure Average Handling Time (AHT), which tracks the average length of calls taken by call centre operators. The idea is usually to reduce the AHT so more calls can be handled and fewer staff are required to operate the call centre. As we’ve observed before, the limitation of this measure is that it encourages call centre operators to do whatever they can to keep calls from going beyond the target AHT. Moreover, if the organisation’s strategy includes a critical success factor focused on customer loyalty, customer intimacy or customer service, then AHT is a poor measure to use because it directly undermines these other objectives.

Another risk with limiting our selection of measures to existing data and legacy measures is it keeps people focused on yesterday’s (or yester-year’s) strategy. The parts of the organisation’s current strategy that are new will go unmeasured. And when something isn’t measured, it’s very hard to keep people focused on it, working toward it, and achieving it. If call centre staff are measured only by AHT rather than how well they solve a customer’s issues the first time the customer calls, then a new strategy to provide rapid solutions to customers will fall on its face.

We know that if a performance measure is going to be used to improve performance and achieve a specific result, the people who produce that result must feel a strong sense of ownership towards that measure. And that feeling comes from hands-on, active involvement in choosing that measure—not from being handed the measure and told to start reporting it and meeting its targets. People might feel quite familiar with and accepting of their legacy performance measures, but it’s not the same as feeling true ownership of them. One of the legacy performance measures in call centres is Cost per Contact, which is the total operating expense of the call centre divided by the number of calls it receives. It’s a very threatening measure for call centre staff because the overwhelming perception is that it can be improved only by reducing the number of call centre operators. Staff are familiar with Cost per Contact. They might accept it as part of the landscape of the organisation. But they don’t feel ownership of a measure that poses a direct threat to their jobs.

When it’s so easy just to keep measuring what’s always been measured, better measures are rarely explored. And unless we explore better measures—measures that are more aligned with a new strategy, measures that don’t drive dysfunctional behaviour, measures with which employees feel engaged- organisations will never start collecting the data they truly do need and will more likely need into the future.

An approach to designing performance measures certainly needs to produce measures that are cost effective and quickly establish a baseline of historic data. But it must also produce measures that will take the organisation toward its vision, not keep it anchored in the past.

Adopting measures from industry benchmarking

There are two types of benchmarking, and we need to define them before we talk about benchmark measures. The first kind is process benchmarking, where organisations learn from best practices in other organisations. This kind of benchmarking uses performance measures only to identify which organisations appear to have the best practices. Most of the emphasis is on learning from those other organisations how they have designed and managed their processes to achieve such great results. One example of process benchmarking is a firearms company benchmarking with a cosmetics company to improve its ammunition shells by learning how the cosmetics company made their lipstick cases so beautiful and shiny. Another is an airline that wanted to improve its maintenance process, so it benchmarked against a NASCAR pit crew.

The second kind of benchmarking is numeric benchmarking, and is often used to obtain ideas on how to measure specific goals or objectives. Numeric benchmarking emphasises the comparison of specific performance measures within an industry, often to rank organisations within that industry and identify best-in-class performers. It can be coupled with process benchmarking, but numeric benchmarking studies often stop at the numeric comparisons and rankings. And benchmark reports like these can inspire some organisations about what to measure. A client of mine used CAPS Research²⁴ benchmarking reports to help identify performance measures for their procurement process. Thankfully they used the metrics from the benchmarking reports as an input into a robust procedure for selecting measures, which they filtered to find the measures best suited to their procurement strategy. They didn’t do what many would: pick and choose any measures that look interesting, or for which they already have the data. Or, worse still, pick every measure from the list.

Benchmarking has some advantages. People feel safe and secure because others have gone before them in considering what might be meaningful to measure for a specific goal or objective. Others have (we assume, but not always correctly) already put a lot of thought into those measures, so why reinvent the wheel? Adopting benchmarked measures also means people can compare the performance of their department, team or organisation with the performance of similar departments, teams, or organisations and get a sense of how good (or how much better) they are compared to others.

Benchmarking is also relatively easy and widely accepted, with many industry associations and research groups coordinating these kinds of activities. And it’s true that when we’ve chosen a measure that others in our industry are already using, it’s easy to justify its use to colleagues and stakeholders.

But benchmarking has disadvantages too, which can significantly limit the value of the measures chosen. There are consequences to adopting a ‘bolt on’ set of performance measures. One of the most important limitations is that benchmarking isn’t always collaborative. A small team is usually tasked with the benchmarking research and choice of measures, so there is little buy-in from others who have to use and meet the standards set by those measures. In fact, there is often deep cynicism about those measures chosen, which results in the kinds of dysfunctional behaviours we hear about all the time:

figures are fudged to make the measures look good

people spend more time complaining about the lack of relevance of the measures than they do putting them to use in improving performance

people fail to implement the measures at all.

An association representing landscape fire management agencies attempted to develop a set of consistent performance measures for everyone to use. But they failed to get those measures endorsed and implemented because they didn’t involve the right people in the dialogue about the results that mattered most and the design of the best measures for those results.

A more subtle but significant limitation of benchmarking is it is driven by convenience and a questionable desire for industry consistency rather than the organisation’s unique strategic direction and the information needed to understand and execute their strategy. No two organisations are exactly alike, have exactly the same strengths and weaknesses, or aspire to exactly the same vision or values. So no two organisations will share exactly the same strategy, or need exactly the same performance measures. Performance measures are how we track our strategy execution and achievement, and are as unique to an organisation as its strategy. So how can a consistent and standard set of industry performance measures ‘bolt on’ to a unique organisation and support its unique strategy?

Benchmarking doesn’t challenge our assumptions either. One of the profound benefits of performance measures is they force us to get very clear about the goals we’re trying to achieve—what they really mean, and what they really look like when they are achieved. Blindly adopting measures just because other organisations are using them robs us of the opportunity to have this dialogue. The procurement team I mentioned earlier had a lot of success with selecting some relevant measures from the CAPS Research metrics. But there were a few results they still needed to design unique measures for because they did not find anything suitable in the benchmark report. For example, part of their procurement strategy was to encourage the organisation to order goods and services using approved contracts with suppliers. They needed to design their own measures²⁵ for this result because there was nothing appropriate in the CAPS benchmark report.

And, as with brainstorming, when we rely on benchmarking it’s too easy to choose measures based on an incorrect assumption about what a goal really means. In fact this is a recurring theme across all of the common approaches people use in selecting measures. They jump to the question ‘When should we measure?’ before they have a sufficiently clear and shared understanding of the results they feel are important to measure.

Looking at a selection of industry-standard performance measures can be useful, but only when people are very clear about the organisation’s strategy, and have a deliberate method to filter a list of potential measures to choose the ones that best align to that strategy.

Measuring what stakeholders want to measure

Often the requirement or expectation to measure certain things is imposed on an organisation by regulators, shareholders, government, industry bodies or other stakeholders. And the requirements of the body mandating the measurements can weigh heavily in the decision about which performance measures the organisation will use.

Government departments typically have a set of performance measures that have been insisted upon by the overarching government body, and that must be reported as part of the budget process. Just about every government department among my clients complains that these measures are too operational, not at all outcome-oriented, and don’t fit with their department’s strategic plan.

There are a few apparent advantages of an organisation measuring only what it’s told or requested to measure by stakeholders. A lot of the hard work of figuring out what stakeholders might want to see in a strategic plan is eliminated. The stakeholders simply state what they want. And when an important stakeholder wants something measured, senior managers are likely to feel a commitment to measure it.

On the downside, people often struggle to retrofit the stakeholder-chosen measures to their department’s or division’s strategy, or try to renegotiate the stakeholder-chosen measures to get a tighter fit. Measures imposed from outside do not replace measures chosen through thoughtful deliberation by the organisation itself. Yes, it is important to report certain measures to meet legal and statutory obligations and the information needs of external stakeholders. But these externally mandated measures are often confused with performance measures and force-fit into strategic plans when they don’t have to be. When providing information is part of meeting legal and statutory obligations, it simply means reporting information like a product for a specific stakeholder group. It’s not performance measurement, for several reasons.

First, external stakeholders have their own unique stakes in the organisation, and few (if any) of them have a clear view of the whole picture the organisation operates in. The stakeholders’ own focus may not be the right focus, or the only focus that matters for the organisation. No single external stakeholder group is in a better position to understand the organisation’s strategic direction than its own senior leadership team. Senior leadership’s responsibility is to set the strategy and make sure it is measured, resourced and executed. In short, performance measurement needs to be driven by the decisions the organisation needs to make for its own purposes, and the information it needs to make those decisions. It should not be driven by the information requirements of one or more stakeholder groups.

When measures are imposed by external stakeholder groups or other entities, people within the organisation usually do not feel ownership of those measures. It’s like the landscape fire indicators example I mentioned earlier: if people aren’t involved in the selection of the measures, they almost always lack a feeling of ownership towards them.

And let’s not ignore the likelihood, given how many people still struggle to find meaningful performance measures, that the stakeholders themselves do not have robust methods of choosing the most meaningful measures. The measures imposed by outside stakeholders might not be very good performance measures, or even performance measures at all.

An approach to designing measures should produce measures that are relevant to the organisation’s strategic direction, which is understood and supported by its key stakeholders. And this approach to designing measures might involve stakeholder representatives in the process. But the process of designing measures can also be used to properly design the statistics or measures stakeholders require for their own purposes and needs. And the statistics and measures stakeholders require may or may not overlap with the organisation’s performance measures.

Outsourcing the selection of measures to experts or consultants

Industry experts, consultants, people with years of experience and self-nominated experts all have an aura of knowledge and wisdom that can make their ideas about what to measure sound more like truth than they actually are. And getting an expert to quickly fill the performance measure column in the business plan is tempting because it requires little effort and doesn’t necessarily cost a lot.

The first time I realised this was a problem was in my early days as a performance measurement practitioner. The organisation I had just started working for had previously hired a large consulting firm to recommend performance measures. This consulting firm conducted a range of interviews with managers and staff throughout the organisation, then trotted back to their offices to produce a report more than an inch thick recommending dozens and dozens (and dozens) of performance measures. The corporate planning manager called me to his office, presented the report to me, and asked what we should do with it. Each page was devoted to a single performance measure, with the name of the measure in very large type and, if I recall correctly, a very rudimentary description of what the measure was. There was no rationale for the measure. There was no alignment of the measures with the corporate strategy, departments goals or business processes. There was no discussion about the relative strengths or weaknesses of each measure. It was simply a metrics shopping list for this organisation’s industry. I’ll bet the consultants used brainstorming and benchmarking to populate that list.

There are clear advantages to hiring the experts to do the legwork and find the best performance measures for an organisation. People within the organisation don’t have to do the hard work, and can keep focused on their ‘real jobs’. Experts can bring new ideas and experience that people within the organisation might not have. Experts also have approaches that have worked for other organisations, which gives their ideas instant credibility.

But as tempting as it might be to outsource the selection of performance measures, the risks are often not worth the supposed convenience. Some people don’t feel confident enough to challenge an expert, even though they have much more intimate knowledge of their own business and its priorities. And in a world where good performance measurement is still a mystery to many, experts can get away with a ‘one size fits all’ set of performance measures for each industry they’ve worked with. Some have their own proprietary methods for developing performance measures, but many of those methods are just variations on brainstorming with a catchy name.

Experts can also assume their clients don’t need to know the thinking behind the measures, so people within the organisation don’t learn how to think more wisely for themselves about the measures. Or the staff and management might not really understand the measures recommended to them, much less the implications for how they will be created, reported and used.

But one of the biggest downsides of hiring experts to select performance measures is those measures will have very low buy-in. Most people are reticent about performance measurement to begin with, and it’s only when they get to be part of the thinking, designing and decision process that they feel any real sense of engagement and ownership. So if you’re looking for someone to tell you what your measures ought to be, don’t ask me.

The approach we use for designing measures needs to produce performance measures we understand, and of which we feel ownership. It needs to let us continue refining and refreshing our selection of measures as our performance and strategic direction change. Selecting performance measures should not be outsourced because it plays such an important role in focusing everyone on what really matters most and giving them feedback to improve what matters most. It’s one of the most strategically important management processes any organisation has. I love what Dean Spitzer, author of Transforming Performance Measurement,²⁶ says on the very first page of his book. (I quoted at the start of Chapter 1 of this book because I think it’s such an important point.)

‘What if I were to tell you that one of the most important keys to your organization’s success can be found in a very unlikely place—a place many of you consider to be complicated, inaccessible, and perhaps even downright boring? What if I were to tell you that this key to success is already one of the most ubiquitous and impactful forces in your organization? It’s there, waiting for you to tap into it.

The key to success is MEASUREMENT.

Measurement done right can transform your organization  .  .  .  measurement is fundamental to high performance, improvement, and, ultimately, success in business, or in any other area of human endeavour.’

Why would we outsource something so vitally important?

Is there a better way to find performance measures?

The reason many people use these quick and easy approaches to select measures is the same reason performance measurement is a dreaded event: people have no idea that designing measures is a process of dialogue about the goals or objectives they want to measure. When performance measures are designed well, the process is a way for people to deeply understand what needs to be better, and to know whether or not those things are getting better (and by how much).

Even if we used a combination of the typical approaches to selecting performance measures to minimise the limitations of individual approaches, we would still fall short of producing a balanced, meaningful and motivating set of performance measures.

Performance measures are comparisons that provide objective evidence of the degree to which a performance result is occurring over time. Because measures are evidence of results, any approach to designing measures has to start with understanding what evidence would convince us the result we’re trying to measure is occurring, and to what extent. We can measure only what we can observe, so observation is the basis for evidence gathering.

We also want our approach to produce performance measures that are meaningful, even for seemingly immeasurable or intangible goals. We want to be sure our measures are few and focused, and that they are not going to add more effort or complexity to reporting. We want the confidence of using measures that have been thoroughly tested for their relevance in tracking the goal or result for which they are selected, and are supported by the people who will have to do the measuring.

A good approach to designing measures will produce measures that encourage learning, knowledge sharing, and collaboration (rather than competition in performance improvement) and assist us in discovering the best business strategies for the organisation to achieve its goals. Our approach should help us find measures that are cost effective to implement from data collection to reporting, and will take the organisation toward its vision, not drag it back into the past or lead it off on a tangent. With a good approach to designing measures, people will feel strong ownership of—and even excitement for—their measures.

The PuMP Designing Measures technique that we’re about to explore is based on five principles:

1. Facilitate the people who will own the measures to develop the measures.

2. Start with the goal, objective or result that we want to measure, and maintain a keen focus on what that result really means while we search for appropriate measures.

3. Consider a range of potential measures, and contrast their relevance to the result and their feasibility to implement before we make a decision about which measure (or measures).

4. Choose the best measures; don’t wait for perfect measures.

5. Be specific in articulating the measure so it’s apparent what is being measured, and how it’s going to be measured.

The approach to designing measures we’re about to learn is quite straightforward and seemingly simple. But its power should not be underestimated. During the 14 years prior to this book’s publication, I helped more than a thousand people use the PuMP approach in their organisations. And each time they’ve been pleasantly surprised how easily it helps them produce valuable performance measures they feel excited about.

This approach to designing measures is great for crafting new measures from scratch, as well as evaluating existing performance measures or measures found through the traditional approaches described at the start of this chapter. It works smoothly in a team setting where several people are collaborating to find performance measures for their own goals, objectives or results. In fact it works best in a team setting because group dialogue makes it easier to explore and test the meaning of a result, to build a fruitful list of potential measures, and to critique them realistically.

The five steps in this PuMP Designing Measures technique are:

Step 3.1: Begin with the result in mind

Step 3.2: Be sensory-specific

Step 3.3: Find potential measures

Step 3.4: Check the bigger picture

Step 3.5: Name the measure(s)

Let’s look at each step in more detail.

The 5 steps of meaningful measure design

Step 3.1: Begin with the end in mind

As the first step in deciding how to measure a result, we write down what the result is so we can keep focusing on it as we consider potential performance measures. This enables us to easily and quickly assess the relative strength or relevance of each potential measure, and avoid jumping to conclusions too soon. We focus on one result at a time.

Step 3.1 is simple: Write down the result for which you currently want to find a measure. Put it at the top of a page or whiteboard so everyone who is collaborating on designing measures can stay focused on it. It’s surprising how readily people will suggest performance measures that are only loosely associated with it, and not directly relevant to it. If we’ve written the result down for all to see, we’ll have a visual anchor to which the group can keep returning. Where do the results come from? They come from the organisation’s strategic or business plan, or ideally from its PuMP Results Map.

Wollondilly Shire is a local government area located in the beautiful New South Wales countryside, about an hour’s drive southwest of Sydney, Australia. The Shire Council offices are based in the little town of Picton, famous (at least in part) for being the location of a scene in The Wolverine, which stars Australian actor Hugh Jackman. My first visit to Picton was to help the management team find measures for the performance results we ‘teased out’ of their strategic goals (PuMP Blueprint Step 2).

One of the performance results was ‘People in the community participate with each other in organised community activities’. Writing this down at the top of our PuMP Designing Measures template (which we will see in its entirety toward the end of this chapter) was all we had to do for Step 3.1: Begin with the end in mind.

Step 3.2: Be sensory-specific

When we have the end in mind, we are ready to get a handle on what we will measure about our result. We need to take care to describe the result as specifically as possible. A good way to be specific is to use sensory language that describes what we observe with our five senses: sight, hearing, touch, taste, and smell. (Sometimes we include a sixth sense, kinaesthetic or whole-body awareness, which encompasses movement.) Read the poem at the start of this chapter once more, the one by Dorothea Mackellar. Notice how she uses words to paint a visual image? If you’re Australian, or have seen pictures of the Australian landscape, this poem will trigger particularly vibrant images in your mind.

To make our performance results sensory-specific, we list the things we and others would see, hear, touch, taste, smell or sense with our whole bodies if our result was happening now. We are looking for the unequivocal, observable or detectable evidence that our result is occurring. We avoid inert ‘weasel words’ that appear so often in goal and objective statements— ‘efficient’, ‘effective’, ‘accountable’, ‘committed’, ‘functional’, ‘productive’, ‘consolidated’, ‘advanced’, ‘reliable’, ‘sustainable’, ‘integrated’ and so on. None of these words can be observed with our senses, and we simply can’t measure a result described using inert language. Measures are based on data we analyse to give us some evidence of the degree to which a particular result is occurring. Evidence is found in the physical world, the world of our five senses. So if a goal or result is to be measurable, then we must be able to describe it in terms of what someone would see, hear, feel (or do), taste, smell, or otherwise experience physically if that result were occurring. For example, a measure of employee absenteeism is based on data about individual employees’ attendance at work. It’s evidence about whether or not we see each employee at work on a given day. Over time, we analyse this data to measure the degree to which employees are available for work.

It can help if we ask questions such as:

What would people see if this result was already achieved?

What would people feel or be able to touch if this result was already achieved?

What would people hear if this result was already achieved?

What would people taste or smell if this result was already achieved? (Granted, most performance results in business do not have a flavour or a smell.)

What would people experience if this result was already achieved?

Asking ourselves these questions when considering the sensory-specific evidence of ‘customers are loyal’ we might produce the following list:

We would see the same customers coming back for our services again and again.

Customers would feel a strong desire to keep using our services again and again.

We would hear customers saying how likely they are to keep using our services.

We would observe in financial reports the continuing revenue earned from each customer over time.

So what is the effect of sensory-based language? We can more easily imagine in our minds what it would be like if a goal or result was happening. We can also more easily pinpoint the kind of evidence that will help convince us about the extent it is happening as time goes by. By thinking about how we can quantify the evidence, we can construct our meaningful measures. You are probably already thinking about potential measures for the sensory-specific evidence of ‘customers are loyal’, aren’t you? Hold your horses and try not to jump ahead too quickly! Step 3.3 is going to guide us in the best way to capture our ideas for potential measures based on our sensory-specific evidence.

In this step of the Measure Design technique, we create a list of the observable or detectable symptoms Sherlock Holmes would snap up as evidence that our result was happening. It’s surprisingly difficult to list the evidence that shows that a result is occurring. People tend to make the same set of mistakes the first time or two until they see how this step helps the next Step (3.3). Many people list things that are not sensory-specific evidence of the result they are designing the measure for.

A spin-off benefit is one kind of evidence people often come up with that is not sensory-specific evidence of a result. Spin-off benefits are descriptions of a different result that would be an effect of the result for which we are designing the measure. For example, ‘increased profit’ could be a spin-off benefit of successfully achieving the goal ‘customers are loyal’. But profit isn’t evidence of loyalty. We can have loyalty without increased profit, and we can have increased profit without loyalty.

Another thing that is not sensory-specific evidence of a result is the use of vague weasel words. Don’t let the weasel back in! We do ourselves no favour at all by writing ‘enhanced customer engagement’ as sensory evidence that ‘customers are loyal’. That phrase is not sensory at all. How can we see, touch, smell or taste engagement, or its enhancement? What do those terms even mean?

A third thing that is not sensory-specific evidence of a result is actions. For the result of ‘customers are loyal’, writing down ‘customer loyalty program’ is most definitely not evidence of how loyal customers are. Yes, creating such a program could help achieve the result. But it is not the result itself.

These types of problems crept into the first Measure Design that Wollondilly Shire Council’s management team did. But they quickly picked up on what good sensory evidence is, as you’ll see in the sensory-specific evidence for ‘People in the community participate with each other in organised community activities’, shown in Table 3.

Step 3.3: Find potential measures

If we do a good job of listing the sensory or observable evidence of our result, we’ll be able to translate the evidence into potential measures that tell us the extent our result is occurring. It is important not to filter out any ideas with thoughts like ‘We don’t have the data for that’ or ‘That’s too hard to measure’. We need to just do one thing at a time, and do it properly. Right now, the thing we’re doing is listing potential measures. We don’t have to sign them off in blood just yet. We’ll make it easier by splitting Step 3.3 into two sub-steps: listing our potential measures, and evaluating our potential measures.

LISTING POTENTIAL MEASURES

As we write down potential measures, we need to be sure we are writing down real measures, and not vague phrases, milestones, actions or data sources. We want to write down sentences expressing comparisons that provide objective evidence of the degree to which each type of sensory-specific evidence is occurring over time. Now might be a good time to revisit the definition of what a performance measure truly is on page 22.

For Wollondilly Shire’s desired result ‘People in the community participate with each other in organised community activities’, they translated their sensory-specific evidence into potential measures as shown in Table 4. There were quite a few potential measures for this result, but that’s not always the case. And there is no correct number of potential measures either. We just list them until we run out of ideas.

Remember, we’re not interested in evaluating these measures yet. It’s much faster if we list the potential measures first, without judgement, and then continue with the second part of Step 3.3 once the list is done. The evaluation will sort out which measures are useless more quickly than having an unstructured discussion as each potential measure is suggested.

TABLE 3. Examples of sensory-specific evidence for community participation
TABLE 3. Examples of sensory-specific evidence for community participation

TABLE 4. List of potential measures for community participation
TABLE 4. List of potential measures for community participation

EVALUATING POTENTIAL MEASURES

After we’ve finished a list of potential measures, we filter them to help the best become obvious. The evaluation is based on asking two questions for each potential measure:

How strong an indicator is this measure of the result?

How feasible will it be to measure this?

To answer the question about the strength of each of our potential measures, we need to read our desired result again. If a measure is strong, then we’d be very convinced our result was happening from that measure alone. If the measure is weak, then we’d imagine our measure could be showing positive changes, but the result itself remains unchanged. For example, our result might be that customers are really happy with our service. And a potential measure might be ‘Number of Customer Complaints’. This is a weak measure because the number of complaints can decrease simply because customers have given up complaining about bad service even as their dissatisfaction increases. Another reasonable scenario is that customers feel much more welcome to offer constructive feedback when they are satisfied with our service, so our measure could look worse even though customer satisfaction (the desired result) is better.

To answer the second question about the feasibility of our potential measures, we need to think about the ease or difficulty of accessing the data we’d need for the measure. High feasibility means that the data is already available, or would cost minimal time or money to get. Low feasibility means the cost of getting the data is too high, and wouldn’t likely be worth it. A measure of ‘Number of Customer Complaints’ is usually quite feasible. Many organisations log complaints in a database, and so all we’d need to do is count them. Measuring customer satisfaction directly might be less feasible if we haven’t already established a routine and statistically valid customer survey.

For both strength and feasibility we can use a 7-point scale, where 1 is extremely low and 7 is extremely high. That is sufficient to sift even long lists of potential measures without being too complex. I find that when I use fewer than 7 points it’s harder to separate the most powerful measures from the others, and we can end up with more measures that we need. And when I use more than 7 points we waste time haggling over whether a measure is an 8 or a 9, and that degree of sensitivity is not worth the extra time.

Wollondilly Shire’s management team quickly reached agreement on the strength and feasibility of their potential measures for ‘People in the community participate with each other in organised community activities’. Table 5 shows the measures they decided were best, based on an evaluation of strength (S) and feasibility (F), with the measures chosen as best indicated by a check mark in the final column.

The table shows that wherever the Wollondilly team gave a measure a strength rating of less than four, they didn’t waste time rating its feasibility. If the strength alone is that low, the measure shouldn’t even be considered a good potential candidate for this result.

Colorado’s Douglas County is another local government organisation that used the PuMP approach to Designing Measures. One of the teams involved in designing the measures deviated from the PuMP steps, and chose to ignore feasibility. This came back to haunt them when they built their performance scorecard, and found gathering the data for their measures far more onerous and time consuming than they’d imagined. This example is a reminder that every step in the PuMP Blueprint is there for a good reason, and is based on lots of experience.

Step 3.4: Check the bigger picture

It’s important to check the bigger picture of what could happen if we measure each particular result. What might the unintended consequences of measuring the result be, both positive and negative? What behaviours might the measures unintentionally encourage? What other areas of performance might be sabotaged, limited or leveraged as this result improves? This is our first chance to change our minds about what to measure for this result, or even whether to measure this result at all.

One of the classic unintended consequences is that if we improve results for which we produced measures in Step 3.3, other measures would be undermined. At Douglas County, in designing a measure for their result ‘reduce costs and eliminate waste’,²⁷ the team identified a negative unintended consequence: ‘Costs may be cut to such an extent that service capacity is negatively affected’ and ‘Certain services that may be outsourced may lower satisfaction levels if performed by non-county staff’. But they also identified a positive unintended consequence of measuring costs and waste. The measure might start a ripple effect of instilling a cost-efficiency mindset in staff: ‘Employees may continue finding ways to reduce costs that continue to reinforce our culture of fiscal responsibility’.

TABLE 5. Evaluating strength and feasibility of measures for community participation
TABLE 5. Evaluating strength and feasibility of measures for community participation

Back to Wollondilly Shire Council. For their chosen measures for the result ‘People in the community participate with each other in organised community activities’, they quickly realised one very important feature about the bigger picture for those measures, shown in Table 6.

TABLE 6. Checking the bigger picture for a community participation measure
TABLE 6. Checking the bigger picture for a community participation measure

Performance measures aren’t limited to monitoring only those results that are totally within our control. In fact, there really isn’t much that is totally within our control. When we’re working in an organisation or business, we will have the most success and the greatest impact when we turn our attention to the results within our circle of influence. The measures we want to avoid are ones addressing results that are only within our circle of concern. The concepts of circle of control, circle of influence, and circle of concern come from Steven Covey’s landmark book The Seven Habits of Highly Effective People²⁸ where he summarises them as follows:

Instead of reacting to or worrying about conditions over which they have little or no control, proactive people focus their time and energy on things they can control. The problems, challenges, and opportunities we face fall into two areas—Circle of Concern and Circle of Influence.

Proactive people focus their efforts on their Circle of Influence. They work on the things they can do something about: health, children, problems at work. Reactive people focus their efforts in the Circle of Concern—things over which they have little or no control: the national debt, terrorism, the weather. Gaining an awareness of the areas in which we expend our energies in is a giant step in becoming proactive.

While we can (and should) measure important things in our circle of control, the measures with the most potent power are the ones that measure results that lie within our circle of influence. Measures of results in our circle of concern, on the other hand, are powerless.

Step 3.5: Name the measure(s)

Naming our performance measures marks the point where we know exactly what we will be measuring. Names should be succinct, informative and deliberate because we need to be able to consistently and easily identify each measure as it is brought to life and used in decision making. There are two things to do. First, we give our measure a brief, three- to five-word unique name. Second, we write a sentence that describes the measure in enough detail that we can start to picture how it would be calculated.

Table 7 shows how the Wollondilly Shire Council team named and described their new performance measures for ‘People in the community participate with each other in organised community activities’.

These measures still need some work to make their calculations and data requirements clear. But that’s what PuMP Blueprint Steps 4 and 5 will help to do. Time is the best way to create your performance measures is one step at a time.

TABLE 7. Naming measures for community participation
TABLE 7. Naming measures for community participation

The full design of the community participation measures

In Table 8 we can see all the steps of the PuMP Designing Measures technique for Wollondilly Shire Council’s result of ‘People in the community participate with each other in organised community activities’.

These completed Measure Designs are a very useful way to keep a record of the thinking that led us to the choice of each of our performance measures. This makes it easier for others to understand why certain measures were chosen and others ignored, which can increase buy-in.

TABLE 8. The full Measure Design for community participation
TABLE 8. The full Measure Design for community participation

Designing Measures using PuMP compared to traditional approaches of selecting measures

The PuMP Designing Measures technique, like the traditional methods, has its advantages and disadvantages. On the plus side, it produces meaningful and quantifiable measures. It tests the rigour in our thinking about our strategy, and ensures tight alignment between our measures and that strategy. It’s very collaborative because it is dialogue-based, resulting in strong ownership of the performance measures it creates. It takes into account the bigger picture, verifying that the measures will be good for the organisation and will not have unintended negative consequences.

On the downside, the PuMP Designing Measures technique is not as easy as traditional methods at first. As with anything new, patience, persistence and practice is required to develop skills in Measure Design. Until people have been through it once or twice, they can become distracted if they forget what step they are doing and why. Designing meaningful measures using PuMP takes time because it asks for open dialogue to build a deeply shared understanding of what is being measured. And it needs some training and practice to ensure that people can use it properly. In addition, using the PuMP method and exploring it in sensory detail can reveal a need to alter our organisational strategy. The sensory-specific step in particular is harder than it first appears. Actually, having written this list of supposed disadvantages, I think deliberate process, open dialogue, developing deeply shared understanding of what we are doing and discovering ways to strengthen our organisational strategies are actually advantages.

Our Measures Teams are probably excited about their new performance measures at this point, and we want to keep that excitement alive by bringing those measures to life and using them. However, we don’t want rush in without the wider ownership of the measures needed for them to be used successfully. So we get together again in the fourth week in our PuMP Pilot for Step 4 in the PuMP Blueprint, where we engage others and solicit some feedback to make sure the measures are meaningful and have the support of those affected by them.

Take Action: Design meaningful performance measures

Once you have some measurable goals or objectives—not weasely, vague ones—you can quickly get started on designing a few meaningful performance measures for them.

Go to the book’s website (www.pumpblueprintbook.com) and download the PuMP Measure Design template.

Chapter summary

We traditionally use methods such as brainstorming and benchmarking to select performance measures without realising these approaches cause many of the problems we experience with our measures. The approach we use determines whether our measures are sensible, quantifiable, aligned with our strategy, and owned by those affected in the organisation.

The five steps of the PuMP Designing Measures technique are:

1. focus on the result we need to measure,

2. describe that result in sensory-specific detail,

3. develop a list of potential measures and short-list them based on their relative strength and feasibility,

4. check to ensure the measures don’t have any unwanted or unintended consequences, and

5. name and describe the selected measures.

To design good measures, we need to sharpen our evidence-based thinking skills, because performance measures are evidence of performance results.

Is your Measures Team ready to take their new measures out into the organisation and build the buy-in and support to get them implemented?

7

Step 4: Building Buy-in to Measures

‘It would be wonderful if the good ideas you champion, on or off the job, could simply stand on their own. But far too often, this is not the case.’

— John P. Kotter and Lorne A. Whitehead from Buy-in: saving your good idea from being shot down

Buy-in is the emotional state in which people feel committed to something. Then they are convinced of its worth for them, and no longer have objections or fears that get in their way of adopting it or contributing their time and energy to make it work. They feel a sense of ownership. They’re engaged.

When we establish strong buy-in to performance measurement, some magic happens. Performance starts improving even before the first improvement initiative is implemented. People have more energy and enthusiasm because they come to work for a reason that is more motivating than getting paid: the impact they have on customers, the community, or even the planet. Performance improvement happens faster when people value measurement because more of their focus is on what really matters most. The ‘us and them’ boundaries between organisational units dissolve, replaced by conversation and collaboration. And strategy comes alive in our day-to-day work instead of existing solely as a collection of platitudes in the organisation’s vision, mission and goal statements.

Most of us want our colleagues and staff to feel this way about performance measurement. But it’s difficult to get people to buy in. Why? I reckon it’s because performance measurement is usually implemented poorly and used for the wrong reasons, so people have been burned by previous bad experiences. Many people don’t just lack buy-in for performance measurement. They resent it, which makes it especially hard to successfully embed good performance management into an organisation or business.

Why do people so desperately want to avoid performance measurement?

Performance measurement has such a terrible stigma. Many people associate it with the inane drudgery of data collection, with pointing fingers or big sticks that beat them when things go wrong. They associate it with the embarrassment or shame of being compared with whoever is performing best this month. The emotions people typically feel in response to performance measurement are frustration, cynicism, defensiveness, anxiety, stress and fear. Sure, some people thrive on competition, comparison, and striving to be on top. But that’s a small minority. Organisational performance will flourish only when the majority feel capable of making a valued difference and are motivated to do so, rather than threatened by the possibility of punishment as a consequence of performance measurement.

What we really want is for measuring performance to be seen as a natural and essential part of work. We want people to associate it with learning more about what works and what doesn’t, with valuable feedback that keeps us on the right track and with continuous improvement of business success. We want people to appreciate that performance measurement is, in its essence, feedback. And feedback is a necessary process for any system to flourish—a fern, a frog, a person, a team, an organisation, a society. We want people to feel curiosity, pride, confidence, anticipation and excitement from measuring performance results that matter.

How can we create this kind of buy-in?

We don’t have to create buy-in. We just have to stop doing things that kill it. Unfortunately, the majority of approaches to performance measurement I’ve seen are very good at killing buy-in. Measuring people in unfair ways, and then rewarding or punishing them based on those measures, definitely kills buy-in. Telling people they have to measure things and leaving it up to them to figure out what the measures should be kills buy-in. Handing people the measures we want them to monitor kills buy-in. Getting experts to tell everyone what to measure kills buy-in. Even ‘consulting’ people for their feedback before implementing a set of measures—a very common attempt to engage people in measurement—kills buy-in. This last one is surprising. You’d think consulting people would get them involved and on board. But all of these approaches, including consultation, deny people the one essential thing they need if they are going to buy in to performance measurement. And it took me a while to work out what that one thing is.

If consultation doesn’t engage people, then what will?

I used to think performance measurement was about the techniques of selecting, collecting, analysing, presenting and interpreting data to inform decision making. It’s true that technique is an important part of the process, but I’ve been excited to learn that performance measurement is mostly about the people.

Since I started this work in the early 1990s, I’ve evolved techniques to help people develop performance measures based on my own experiences of trying something, failing miserably, trying something different, seeing an improvement, and trying some more. (This is what we call ‘action learning’.) One thing I’ve consistently accomplished, beyond developing good performance measures, is buy-in from people at all levels in each organisation. Buy-in wasn’t deliberately designed into the PuMP Blueprint, and so accomplishing it was a pleasant surprise that made me look for the reason. What I’ve learned is that getting buy-in is not necessarily the product of a strategy specifically aimed at getting buy-in. The high rates of buy-in that PuMP achieves are the result of getting out of the way of the organic emergence of buy-in. Let me show you what I mean.

At the time I was first learning about performance measurement I was working for Kevin, the quality manager at Queensland Rail. Kevin was a big believer in what he called ‘soft skills’—the skills of understanding people, accepting and valuing differences, communicating in the language of the listener, and engaging people in change. Without realising it at first, the soft skills Kevin opened my eyes to permeated my approach to performance measurement, and their influence has been profound. Each technique that emerged from my action learning approach to developing PuMP was easy for people to understand, engaged their enthusiasm, held the space for them to have the kinds of dialogue they did not usually get to have and found to be invaluable, allowed them to make the decisions, and helped them create performance measures about which they felt excited. Ease, fascination, dialogue, autonomy and success create an environment in which buy-in naturally emerges.

The PuMP Blueprint creates this environment from the beginning, which is why buy-in is not a separate step in the process. Instead it is woven organically into the techniques used for each step of the PuMP Blueprint.

Here are the specific ways PuMP creates an atmosphere that helps buy-in emerge naturally.

Don’t educate the team; hold the space for their discovery

We start the performance measurement journey by respecting what Measures Team members already know, and we allow new learning to emerge. If team members feel apprehensive or self-conscious, or have divergent views about performance measurement, sharing and acknowledging these views is the essential first step toward a new shared vision of performance measurement.

Our job as their performance measurement facilitators is to first help them see what they have been doing (and are still doing) more clearly. Then, as they start asking new questions, we can offer them suggestions to start a new dialogue. New dialogue about what good performance measurement means is what Step 1 in the PuMP Blueprint (the PuMP Diagnostic) can trigger.

Don’t dictate the team’s goals; help the team articulate them

Our job is to help our Measures Teams explore and decide how their departments, processes or jobs most affect and relate to the goals and purpose of the organisation. If they build their own line of sight from the activities they perform through to the ultimate success outcomes for the organisation, they will get a much deeper understanding of the corporate direction and a greater sense of meaning in their own work. It also deepens their buy-in (especially for their first experience of performance measurement) if we can start our Measures Teams on goals that are important to them.

If the team’s goals have been cascaded to them via the corporate planning process, it’s still very worthwhile to give the team some time to talk about what those goals mean to them and to freely express those goals in their own words. As they do this, they find their own direct connection to those goals, rather than being in a position of having the goals handed to them from outside. Connecting to the corporate goals is exactly what Step 2 in the PuMP Blueprint, Mapping Measurable Results, will help the team to do.

Don’t give the team their measures; teach them how to fish

Our role is to show the Measures Team how to design measures that provide the best evidence of their results. They are the experts in what they do, and in understanding the results they produce. Although hunting around for existing performance measures related to the results the team owns might save time, the team will likely feel that someone else’s measures are trivial, too generic, or just not quite right. And that will stop them from owning the measures. Designing their own set of performance measures, and being in control of how many they have, is much more exciting.

So we offer the Designing Measures technique from Step 3 in the PuMP Blueprint as a tool for the Measures Team to design their own performance measures. It works on the principle expressed in the famous quotation that if you give a man a fish you feed him for a day, but if you teach him how to fish you feed him for a lifetime. The ownership that will result when the team designs their own measures will be far more valuable than the sophistication of the measures they create. With ownership, the team becomes more confident about measuring performance, and motivated to keep honing their skill at choosing and using better and better measures. If we push them to come up with the ‘right’ measures too soon, we squelch their ownership and confidence and end up with measures that might be great in theory, but no one wants to implement or use.

Don’t judge the measures; collaborate to improve them

We encourage the Measures Team to invite feedback from their stakeholders to test and fine-tune their measures. The team’s customers, advisors and partners can collectively view the measures from an holistic perspective, and offer unbiased and complete feedback about the measures’ relevance and practicality. This communication brings the team closer to their stakeholders, enables the team to better understand stakeholder needs, and helps the stakeholders to buy in.

PuMP Blueprint Step 4, Building Buy-in to Measures, is about the Measure Gallery technique—the ideal forum for open, voluntary and supportive collaboration between the Measures Team and their stakeholders. The process produces great feedback that the Measures Team will trust and value to improve the measures, and helps the Measures Team reinforce their own knowledge about measuring performance through sharing it with others. We’ll look more closely at the Measure Gallery after we review the other ways PuMP naturally creates buy-in.

Don’t do it for them; let the team implement their own measures

We need to put the power to bring the measures to life in the hands of the Measures Team. We help the team to see they are not restricted to existing data, don’t have to spend lots of effort collecting new data, can discover very simple ways to bring their measures to life, and can decide not to measure something if it’s not worth the effort.

Step 5 of the PuMP Blueprint, Implementing Measures, gives the Measures Team a tool to get to know their measures intimately. In the process, they will often realise their measures are easier to implement than it first seemed, and begin to more tangibly appreciate how valuable those measures will be. The team will truly be the creators and owners of the measures they bring into the world.

Don’t trample the team’s creative energy

The sixth step in the PuMP Blueprint, Reporting Performance Measures, balances must-have report design features with enough creative license to let the Measures Team have some fun presenting their performance measures, and make the report useful and engaging.

It is important to give the Measures Team some guidelines to inspire their performance report design, and help them decide the types of supplementary information they need to include to make the report usable. There might be some corporate reporting standards to follow. But it’s important for the team to be able to make the report into a tool to serve their own decision making and improvement.

Don’t dumb down the measures; trust the team’s innate numeracy skills

As facilitators, we show the Measures Team the pros and cons of different approaches to interpreting and responding to their measures. Ideally, we do this by using data or measures with which they are familiar. When they see how easy it is to misinterpret data depending on how the data are presented, it won’t matter whether they passed high school mathematics or not. They will realise the power of the right presentation.

Step 7 in the PuMP Blueprint, Interpreting Signals from Measures, will give them a simple, powerful tool that everyone understands when they are involved in applying it to their own performance measures. The renewed confidence they will have in being able to use their measures in a numerically valid way will go a long way toward motivating them to use data to support their performance improvement efforts.

Don’t blame people for performance; let them fix the causes

We want to allow the Measures Team to continually learn how they can improve their results, what the root causes of results are, and where the points of highest leverage are. So we have regular conversations about what their measures are saying to them about the design of their business processes and workflows, and how to link their ideas for improvements and their targets to the corporate goals.

When the team knows how to use their measures in the way that Step 8 in the PuMP Blueprint (Reaching Performance Targets) describes, our conversations with the team about improving performance will focus on root cause analysis and process improvement or redesign rather than on blaming people or circumstances or making excuses.

Add choice; never take choice away

I believe buy-in is a natural product of respecting people, giving or showing them more 1choices than they thought they had, and not getting in the way of their making their own decisions and contributing. Of course we need to be sensitive to the fact that performance measurement is an organisation-wide system, and each team is only a part of that system. But the trade-off should be biased more toward the team’s buy-in than toward sophistication of the measures. We can improve the sophistication of the measures on a foundation of buy-in more easily than getting buy-in to a suite of already sophisticated measures.

There are many other ways to create the conditions in which buy-in to performance measurement blossoms in its own way. Asking more questions and making fewer statements. Believing that people largely want to contribute rather than needing to be controlled. Celebrating successes as well as celebrating failures because they are valuable learning. The common thread is they all make it safe for people to come voluntarily to performance measurement and to make it meaningful for themselves.

So building buy-in for our Measures Teams will be a natural by-product of implementing the PuMP Blueprint in the right spirit. But we’ll also need to broaden and invite buy-in from people outside the Measures Team: colleagues, managers and other stakeholders whose lives are in some way affected by the measures being developed.

For example, we need buy-in from the people working in the same department, business process or business unit as the Measures Team members. It’s too unwieldy for the whole department, business unit or process team to sit together to design measures. Unless we are part of a small business unit with only four or five people, it’s just not practical to have everyone at a meeting where we are mapping results or designing measures. But everyone still needs to feel engaged and buy in to the performance measures.

Other people, such as regulators, executives, customers, suppliers, strategic partners and the people who are going to collect the data for the measures might need to buy in to the measures we are developing if those measures are going to be a success. Any person or group who in some way affects (or will be affected by) the measures needs an opportunity to buy in.

That is what Step 4 in the PuMP Blueprint, the Measure Gallery technique, does. But this effect of the Measure Gallery was a happy accident. I didn’t set out for the Measure Gallery to achieve this.

How to use a Measure Gallery to broaden buy-in

It was a medium-sized meeting room, maybe eight metres by four. The meeting table was still there, filling the centre of the room and adorned with a platter of cookies and a couple of bowls of chocolates, but the chairs were all gone. On the walls were collections of A2-sized sheets of paper, with a Results Map, several Measure Designs, and brief bullet-point instructions duplicated on three of the walls. An A2 poster was stuck with Blu-Tack to the door of the meeting room that read ‘Welcome to our Measure Gallery’. Near the door was a table with a pile of pens and a pile of sticky notes. It was 9 o’clock on a Thursday morning in March, and Darren, John, Sean and I were nervously awaiting our first Measure Gallery visitor.

We had spent the previous few weeks together, working through the first three steps of the PuMP Blueprint to create a set of performance measures for the Investment and Portfolio Services (I&PS) Division of a government-owned investment corporation. This was the first-ever Measure Gallery I had facilitated, and I wasn’t certain how it would work out. But before long people were steadily streaming into the meeting room. Darren, John and Sean welcomed them, explained to them what a Results Map was and how the measures were designed, and handed them a pen and some sticky notes to let them take a closer look and give feedback on the measures if they wished to do so.

A group of visitors started to crowd around Darren as he explained what the Results Map had meant to his team, how it focused them on why they really come to work each day, and how he carried a copy around in his pocket all the time because it helped him explain his priorities to the customers and partners he talked with in daily meetings. And as the day progressed, John earned the nickname ‘Superguide’ for the ease and enthusiasm with which he welcomed visitors and explained how we would like them to contribute. When the Measure Gallery wrapped up at around 4pm, more than 50 of the team’s colleagues (including some senior managers) had contributed ideas, suggestions and congratulations for the I&PS team’s new results focus and meaningful performance measures.

The Measure Gallery turned out to be a great way of getting a lot of feedback very quickly with little disruption to people’s workdays. But the real pearl was that it created the space for people to engage with performance measures in a way they never had before. Its casual and welcoming atmosphere, and the opportunity for natural and spontaneous dialogue, gave them permission to be themselves, contribute what they wanted to, and have their questions and feedback respected. All of this helped them to buy in.

Designing a Measure Gallery can be fun. But there are some very important lines we cannot cross or it simply won’t be a Measure Gallery any more, and won’t have the awesome impact that it should. In fact, we can do more damage to buy-in than good. And they are essential design features to follow when we set up a Measure Gallery.

The design essentials are distilled into the five steps to follow for successful PuMP Measure Galleries:

Step 4.1: Who needs to buy in?

Step 4.2: Invite; don’t mandate

Step 4.3: Allow space for dialogue

Step 4.4: The right people will come

Step 4.5: Gather feedback and use it

Let’s explore what each of these design essentials is about, and how we can put them in place for our Measure Galleries.

The 5 steps for designing Measure Galleries

Step 4.1: Who needs to buy in?

Who are the people with a stake in the measures that the Measures Team is developing? Who are the colleagues, managers, people in other parts of the organisation, people outside the organisation, who might have some kind of stake in what this team does and what it measures? We need to make a list of these people as if we were creating the guest list for a party and want them to be there to celebrate with us.

One way to ruin a Measure Gallery is to ask too few people to come. We need to think of it as a friendly street party, not an intimate VIP dinner party. We will naturally feel some concern about things like getting negative feedback, everyone expecting their feedback will be acted on, or fitting everyone into the Measure Gallery room. But we let those worries go for the time being and simply focus on figuring out whose buy-in we want to nurture and whose feedback should be welcomed and heard.

Dealer Solutions is a small business that specialises in helping Australian motor dealers use new technology to make day-to-day business easy to manage. Dealer Solutions offer lots of time-saving business solutions to the automotive industry, such as computerising vehicle registration, ownership transfer and used car inventory tracking. Sarah is the Project Coordinator at Dealer Solutions, and reports to the Chief Operating Officer, Damien. Both Sarah and Damien took PuMP Blueprint training—Damien came to a workshop, while Sarah did the online program. With Damien’s support, Sarah was charged with facilitating the nine departments of Dealer Solutions to develop performance measures aligned with the company’s strategic direction.

A big part of Sarah’s focus was to get every department measuring a few important results within a year. They really wanted to get performance measurement incorporated into the way they operated as a company. Culture change was at least as important as measuring the right things.

So the focus for each Measure Gallery was to keep nurturing the measurement culture by having members of other departments come to each others’ galleries and discuss the results, the measures, and the rationale for choosing both. They also invited the CEO, Chris, to every Measure Gallery.

The Dealer Solutions departments didn’t invite external customers, suppliers or other partners to their Measure Galleries on this first round. As it was her first time facilitating PuMP (and essentially facilitating nine PuMP Pilots in parallel), Sarah was mindful of keeping it as easy as possible. For our PuMP Pilot, that’s a wise thing to do. When we implement PuMP more broadly later on we can include many more stakeholders on the Measure Gallery guest list. By contrast, when a division of the Queensland Transport Department held their first Measure Gallery for several teams simultaneously they invited a wide variety of stakeholders from other transport organisations. So along with their own employees and managers they had representatives from the railways, regulatory bodies and the government. Around 200 people visited that Measure Gallery in one day, and it was too overwhelming for everyone. The choice of how many people to invite should be made consciously, taking into account who the key stakeholders are, whose presence will help nurture the measurement process, and how many people will be enough (or at least not too many).

This brings us to the next opportunity for us to ruin a Measure Gallery and sabotage the buy-in: making it compulsory.

Step 4.2: Invite; don’t mandate

In our haste and eagerness to get people involved in measuring performance, we can easily fall into typical organisational or bureaucratic protocol and schedule a meeting. But protocol makes people feel a sense of obligation or expectation to attend, even if they don’t want to or don’t have the time. And the stiflingly boring atmosphere of all the meetings they’ve attended in the past will trigger them to attend in a state of compliance—and impatience for it to be over and done with.

Obligation, expectation and impatience get in the way of buy-in because they create a feeling of being ‘pushed’ into measurement. True buy-in can happen only when people feel ‘pulled’, compelled or drawn toward measurement.

If we summon stakeholders to our Measure Gallery by saying ‘You are all requested to come to this Measure Gallery and give us your feedback on our measures’, we will be in trouble. We can’t make a Measure Gallery compulsory. Doing so takes away the intrigue, fascination and, most importantly, the choice. People choose to buy in or to not buy in. And they have every right to make that choice by themselves.

But if we invite stakeholders, we exude a different energy. By making it clear that attendance at the Measure Gallery completely voluntary, and people can come when they like and stay for as little or as long as they like, we stimulate a feeling of mystique and intrigue. We want to welcome people with an invitation, not an agenda. The invitation would be similar to a party invitation, and could say something like this:

We’ve been developing some performance measures. We’ve figured out our priority goals and come up with some measures that we feel excited about

We think you have a stake in these measures, so we are putting our results and measures on display for a day to make it easy and quick for you to tell us what you think.

You are personally invited to come to this room on Tuesday the 6th, any time between 9am and 5pm.

We’ll have some coffee, cookies and cake. Please join us for a quick chat. Stay for five minutes, an hour, or whatever suits you. It would be great to have your feedback if you’re happy to give it.

Here’s the very simple Measure Gallery invitation that the Online Marketing department of Dealer Solutions sent out to their colleagues:

Good Morning everyone,

The measures that were created for Online Marketing Team are up in the corner meeting room now, and will be until 12:00pm.

Tim, Tash and Kayla invite all of you to come and take a look at the results, and at the measures that have been created, and enjoy a party pie with your choice of Tomato Sauce or Ketchup.

Everyone is welcome to leave any comments or thoughts about the results or the maps, via the post it notes supplied. Any ideas on improvements or thoughts about the maps or measures are what this whole process is all about, so feel free to stick any thoughts up on the wall surrounding the other documents.

I look forward to seeing you there!

There might be one exception to this recommendation to invite rather than mandate. It’s something I learned from a colleague, Mark, who has led many PuMP implementations. Mark will book about 30 minutes in the diaries of executives to attend the Measure Gallery. He does this because executives generally have their diaries quite full with appointments, and they are used to flowing with what’s in those diaries. So they turn up at the Measure Gallery, and because it is so different from their typical meetings, it makes a remarkable impact and they get engaged.

Of course, you’ll need a venue for your Measure Gallery, which needs to be set up in a very specific way. The room layout can make or break the success of a Measure Gallery, creating the conditions that will facilitate or sabotage buy-in.

Step 4.3: Space for dialogue

Death by PowerPoint is no joke. When you give a presentation to suggest a suite of performance measures to people their engagement dies, along with any chance they will feel a personal sense of ownership of those measures. Dialogue is the opposite of a presentation; it nurtures buy-in. In a dialogue, people aren’t expected to passively accept a set of measures. Instead they’re encouraged to talk about what performance measurement really means, which performance results are really important to measure, and which measures are useful (or not) for tracking those results.

When we set up a Measure Gallery, we want to avoid creating the kind of bureaucratic trappings that sabotages buy-in. So we don’t present PowerPoints. Instead, we give plenty of time and space for spontaneous and unforced dialogue to flourish naturally. We encourage people to stand and walk around rather than sit. We encourage them to ask questions and chat with anyone they want to, whenever they want to. We hang our Results Maps and Measure Designs on walls in a meeting room and invite people to come by, whenever they like and for as little or as long as they like, to discuss the measures and share their ideas and feedback if they wish. It’s ideal to use a large conference room and remove the chairs so people will move freely, which means they’ll think and discuss more freely.

Most of the Measure Galleries for the departments of Dealer Solutions were held in a small break-out space in their Brisbane office. One wall was entirely window. Although the space was small it was sparsely furnished, with only a sofa. The photography department’s Measure Gallery (shown in Figure 9) shows how simply it was laid out, with the Results Map and Measure Designs hung on the window-wall and treats shared to entice people to visit and as a thank-you for their feedback (Figure 10).

One of the fears Measure Teams have about their first Measure Gallery is that no-one will turn up. Sometimes this fear can cause them to inadvertently sabotage the buy-in by cancelling or delaying their Measure Gallery, or coercing people to visit for the wrong reasons. If we put on the Measure Gallery with confidence, and follow the suggestions above, the right people will come.

Step 4.4: The right people will come

We will get more buy-in if three people turn up to our Measure Gallery because they want to than if 100 turn up because they were told to.

It’s okay if not everyone wants to be involved, and it’s absolutely okay to go ahead with a Measure Gallery even if only a few express interest in attending. The other more reticent invitees might need to see how things pan out before they feel they won’t waste time and energy if they get involved. Or they might feel a bit weird going outside the comfort zone of traditional meetings and presentations if they have never heard of a Measure Gallery before. My clients and I often see that after the Measure Gallery has been open for an hour or two, the early visitors go back to their desks or work areas and tell others about it, and that generates more interest and more visitors.

Figure 9
FIGURE 9. Dealer Solutions photography department’s Results Map and Measure Designs at their Measure Gallery

Figure 10
FIGURE 10. Tempting treats to entice and thank Dealer Solutions’ Measure Gallery visitors

This was the case for a federal government agency regulating sports. The executive team developed the pink and green level in their Results Map from their corporate goals, and then designed measures for those results. They scheduled their Measure Gallery to be open for one day, and invited only the organisation’s middle managers to visit. Overwhelmingly throughout that day, the middle managers recommended that all staff have the opportunity to see the Measure Gallery. So in the end it was left open for an entire week, and every employee was invited to visit, ask managers questions about the results and measures, and start thinking about how their work would contribute to the strategic direction. It turned out to be an important part of cascading strategy in this organisation.

I like the title of this step for designing Measure Galleries: the right people will come. It’s borrowed from Harrison Owen’s four principles of Open Space Technology.²⁹ Here’s what he says about it:

The first principle, Whoever comes is the right people, reminds people, particularly conveners of groups, that it is not how many people come, or even who comes (in the sense of status or position) that counts, rather it is the quality of the interaction and conversation that make the difference. For good conversation you only need one other person who shares your passion.

Owen also discusses what to do if no-one turns up (possible, but unlikely). I think his advice is perfect if no one turns up to a Measure Gallery:

 .  .  .  it may be a great idea but at the wrong time. In [any] case, you are getting some useful information. There is, however, another possibility. It is a great idea, at precisely the right time, and you are the only person competent to deal with it.

We can always walk around the office and remind everyone there’s free coffee and cake in the conference room. Many will come for the goodies, and a few might even stay for the gallery. But almost everyone I know who has dared to host a Measure Gallery has gotten plenty of visitors who end up raving about the time well spent.

Great signs that we have the right people at our Measure Gallery include:

visitors staying longer than they indicated they would

visitors leaving and coming back with their colleagues

small groups of visitors in conversation about the results and measures

small groups of visitors in conversation about their own potential results and measures

enthusiastic comments to us about how worthwhile visitors found the gallery

lots of great feedback on sticky notes (or a Visitor’s Book if you use one)

Varying numbers of employees visited each Measure Gallery held at Dealer Solutions. Chris the CEO also made a point of visiting. He came to every gallery if he was in the office on that day, and when he couldn’t he asked for copies of the measures and Results Maps so he could look at them on his own time and give his feedback to the teams. He was really committed to making sure the results and the measures not only made sense and were going to be valuable, but also were ones the teams themselves stood by and understood. If Chris had been the only visitor to each department’s gallery, the gallery would still have been a worthwhile use of time. Just getting the team out of their usual workspace to talk one-to-one with their CEO about their priority results and measures for a short time is powerfully refreshing and focusing.

Step 4.5: Gather feedback and use it

As we welcome people who visit our Measure Galleries, we ask for their honest and constructive feedback about the performance measures. And we really mean it. We make it clear that nothing is set in concrete, and everyone’s ideas will be carefully discussed by the Measures Team. We encourage Measure Gallery visitors to ask us questions, chat with each other, share their ideas—whatever they have the time and inclination to do.

While gathering feedback through a Measure Gallery can happen in different ways, it is logistically easier if people write down their suggestions and ideas on sticky notes and attach them to the hanging Results Maps and Measure Designs. Afterward, we collate all the feedback and systematically review it with the Measures Team, fine-tuning the Results Map and Measure Designs using the feedback that most resonates.

The Dealer Solutions Measure Galleries gathered feedback using sticky notes, which the teams later used to review their results and measures (Figure 11). But the Measures Teams also captured ideas through the conversations they had with visitors, including Chris the CEO. Chris asked specific questions about the teams’ choice of results: why they chose them, and how they linked to the company direction. He noticed and suggested some missing links between team results and company results, and suggested potentially missing results, too.

A Measure Gallery is very different from the usual report that gets sent around to stakeholders containing a list of measures asking for feedback or a sign-off. This approach looks and feels like a cursory attempt to consult and ultimately ignore people’s input. By contrast the Measure Gallery is a social process. It acknowledges that people are people (not email addresses), and that buy-in is not something we can get, but rather something we have to let happen by removing the typical obstacles.

Figure 11
FIGURE 11. Feedback at the Measure Gallery is gathered on sticky notes

Certainly some fine-tuning of the Results Map and Measure Designs should happen after the Measure Gallery, and after we close the Measure Gallery doors is a good time for the team to sit and discuss the feedback. Not all the feedback will be acted on and, as mentioned in Step 4.1, it’s important not to give to Measure Gallery visitors the impression that it will all be acted on. But visitors do want to know what we do with their feedback, and there are a few ways to let them know. One way is to simply write a summary of the changes we made to our Results Map and Measure Designs in light of the collective feedback we received and share this via email, along with a thank you note to everyone who visited the gallery. Another way, and this is often suggested by visitors to the Measure Gallery, is to hold another gallery after the new measures are implemented so everyone can see the final result. It’s also a great way to celebrate our Measures Team’s success.

So it’s also important to keep the momentum going so the new measures can be implemented quickly and the Measures Team’s energy and excitement stays high. We take them to Step 5 in the PuMP Blueprint next week in their PuMP Pilot, where they’ll create the action plans to bring their new measures to life.

Take Action: Engage your colleagues with a Measure Gallery

Even if you have only a handful of performance results in your Results Map, and a handful of performance measures designed for those results, you have enough to at least run a pilot-test Measure Gallery.

Go to the book’s website (www.pumpblueprintbook.com) and download the PuMP Measure Gallery instruction checklist.

Chapter summary

Buy-in to performance measurement happens when we get the obstacles to buy-in out of the way and create the conditions that allow buy-in to develop naturally. Obstacles include using measures for the wrong purpose, superficially consulting people, or making involvement mandatory. Buy-in is something that has to be felt, not forced.

Buy-in is a natural by-product of the PuMP Blueprint techniques because they are easy to use, they are engaging, they encourage dialogue, they put the decision making into the hands of the measure owners, and they work. These factors create a great environment for buy-in to flourish.

A PuMP Measure Gallery is a great opportunity to broaden the buy-in beyond the Measures Team. It’s an organic, welcoming space where people can give as little or as much time and energy as they wish to learn more about the team’s new measures and share their own ideas and suggestions. The Measure Gallery has to be designed in a very specific way to nurture buy-in. A poorly designed Measure Gallery can undermine or destroy buy-in.

Now that the Measures Team has taken their measures out for a spin to get feedback from colleagues, improved the measures, and increased the buy-in, it’s time to start bringing those measures to life.

8

Step 5: Implementing Measures

‘You should be able to figure out the methods used to create any data. If the documentation is not up to this task, you should regard the data with extreme suspicion. Don’t ever use data unless you know how they were derived, you trust the cited sources, and you agree with the stated methods.’

—Jon Koomey, author of Turning Numbers into Knowledge

In my early years as a performance measurement practitioner, before Kaplan and Norton’s Balanced Scorecard, organisations were only just starting to realise they needed more information about performance than the traditional profit-and-loss statement and balance sheet. In those days, the most common way to choose performance measures was to hire a big consulting firm to research and write a ‘this is what you need to measure’ report.

One of the biggest influences on my personal path with performance measurement was an experience I had repeatedly with these types of reports while working at Queensland Rail during the 1990s. Time and time again I’d see these expensive, inch-thick consultant reports recommending what the company should measure. I would ask the manager who had commissioned the report what she had done with the measures recommended in it, and she would sheepishly reply, ‘Nothing’. That’s right. The report just sat on her shelf and collected dust. There were so many measures that she felt overwhelmed, and not a single one of them was linked to the company’s strategy or business processes. So where should she start? Even if she had sifted through the report to identify measures that seemed relevant to her department, she had no idea about how to take the next step to bring those measures to life.

It would be an understatement to say that half the organisations I have worked with have performance measures that are listed for their business goals and objectives, but have never been reported or used to monitor and improve performance. In other words, these organisations have never had the information those measures could provide as feedback or evidence to evaluate the progress of those goals and objectives. The organisations took the time to choose the measures during the planning process, but not to bring those measures to life to help with executing strategy and achieving goals.

Most performance measurement activity in organisations consists of brainstorming measures for the performance measure column in the business plan. Then, managers fail to give the attention to the logical next step to implement those measures, which is essential if their reasons for choosing measures were anything more than just to appear accountable to their goals and objectives. If managers expect their performance measures to inform their strategy execution and drive true performance improvement, they certainly have to do more to deliver these measures into the world. Performance measures, like babies, are most definitely not delivered by the stork³⁰. As with babies, once we’ve conceived our performance measures, we cannot avoid the labour needed to bring them into the world. But in the case of performance measures, the labour needn’t be excessive or painful if we know how to avoid the common pitfalls that results in wasted time and effort.

The sources of greatest waste in the performance measurement process

A big reason for the ambiguity in how to effectively implement our measures is most decision makers (the people who choose the measures in the first place) are ill-informed about the activities that bring measures to life. As a result, decision makers often don’t have a realistic concept of the processes of data collection, data capture, data analysis and performance reporting involved—or even a realistic understanding of how to interpret and use measures in decision making. Lacking a working knowledge of the performance measurement process, they fail to specify the nitty-gritty of calculating and implementing their measures.

For example, many managers want measures of customer satisfaction without realising just how much planning and effort are required to design and administer a valid way of collecting and analysing customer data to produce such measures. So managers often resort to counting complaints instead, which is not an objective gauge of customer satisfaction. Or they base their measures on data from voluntary email surveys that have low response rates, so the data aren’t reliable. And because the responses are heavily biased, reflecting the input of customers who have strong opinions (and access to email), the data aren’t representative either. We can’t make good decisions about how to improve organisational performance when our data lead us astray.

In general, a lot of effort is wasted bringing performance measures to life, either in the time spent selecting measures that are never brought to life, or in the time spent bringing measures to life in the wrong way. So the labour of bringing many measures into the world is excessive, and far more painful than it needs to be. And as we have all no doubt experienced, this breeds cynicism, overwhelm, disdain, and disengagement from the process of measuring anything, let alone what matters.

The main symptoms of waste in a performance measurement process include:

Arguments about data or measure validity dominate performance review meetings, and delay and derail decisions about how to improve performance.

Measures misinform and mislead decisions because the wrong calculation or analysis is used, resulting in failed or unnecessary performance improvement initiatives.

There are too many conflicting versions of the truth, which result from duplication and lack of discipline in performance reporting processes, and cause confusion and cynicism about the value of measures.

We react to invalid signals while missing important signals in the performance data and measures, which leads us to use precious resources on problems that don’t exist, and not on nipping real potential problems in the bud.

No-one is accountable or responsible for specific performance results, so responses to important performance measure signals are delayed, ad hoc, and not followed through to successful improvement.

Performance doesn’t get improved because it’s not clear to everyone what exactly good and unacceptable performance look like.

We’ll no doubt recognise several of these symptoms in our own organisations or businesses as we delve deeper into how they tend to manifest.

Arguments about data or measure validity

Here is a scene I’ve observed many times. The management team sit around the monthly performance report, debating. They’re not debating the interpretation of the trends or signals in the data. They’re not debating the causes at the root of those trends or signals. And they’re not debating the pros and cons of potential performance improvement solutions. No, they’re debating whether or not a measure or its underlying data have enough integrity to even consider using it. Management meetings are not the time to be debating data integrity. That should have been dealt with before the measures were even added to the performance report or dashboard.

We depend on high-quality data and information as a stable foundation for our decision making. No argument there. Decision making usually involves responding to something. We need our data to validly describe the issue to which we are responding so that we can choose the best response. But no performance measure will ever have 100% integrity, nor should it. The cost of 100% integrity would be tremendous, and would most definitely outweigh the benefit that could come from using a particular measure. So it’s important to understand what level of integrity a performance measure actually needs if it is to guide a good decision without costing the earth.

The integrity of any measure depends on five qualities, which I learned about in my early years as a research statistician.

First, the data must be relevant. The data we select for our measures need to be directly appropriate to the purpose of the performance measure for which we select those data. But too often when we are rushed to get our performance measures reported, we choose data that are available, convenient or seem interesting, and we don’t pay enough attention to whether these data are relevant to our measure. Here’s an example. Employee Engagement might be a measure we choose to monitor how engaged employees feel in their work. If we were to look for data to implement this measure, we may find the only data we capture is in an employee survey where we ask employees to rate their satisfaction with the company on a 10-point scale. We might be tempted to compute the values of Employee Engagement as the average of the employee satisfaction ratings, but that would be a mistake. Those data are not telling us about engagement, so they are not relevant for this measure.

Second, the data must be representative. The data we collect must be observable events or characteristics that describe the full scope of what our performance measure is supposed to be measuring. This means that the data are unbiased. Too often our performance measures simply highlight what the ‘squeaky wheels’ have to say, drowning out the valid, important and balancing views of the ‘well-oiled wheels’ and all the wheels in between. For example, a voluntary survey about how engaged employees are will likely result in mostly those people with strong opinions responding. That will skew our understanding of the extent of engagement employees have.

Third, the data must be reliable. Enough data need to be carefully collected to give us a precise enough measure of what’s going on. Would we rely on one employee’s rating of his engagement to draw conclusions about overall employee engagement? What about five employees? How many employees would we need to ask to get a precise enough estimate of overall employee engagement? Of course, that’s a rhetorical question. Survey statisticians have the methods to work out the smallest sample size we can get away with and still get a very reliable estimate of our overall employee engagement. I ask it just to make the point that we should think about data reliability, and whether we use samples or entire populations as the source of our data.

Fourth, the data need to be readable, which means clearly defined, unambiguous, easy to organise for analysis, and expressed in a language that makes sense to those who will collect, analyse and use the data. If the data are not readable, they can easily be misinterpreted or misused. It won’t matter how relevant, representative, or reliable the data are. For example, an employee engagement survey question like ‘Overall, how much do you enjoy your job?’ will be very readable if employees are given a rating scale that is clearly labelled—say from 0 (not at all) to 10 (completely). But if the question is left open-ended for employees to write comments describing their level of enjoyment, the data would be very hard to analyse and turn into an employee engagement measure.

Fifth, and probably most important, the data should be realistic or resource-effective. It’s sensible to trade off the degree to which our data are relevant, representative, reliable and readable against the level of resources we will need to invest to make it so. We need to be sure that the value we get from using our data is greater than the effort we invested in getting those data. Let’s consider two scenarios. The first is using a sample size of 500 employees to know with 99% confidence that the percentage of engaged employees is between 78.1% and 78.3%. The second scenario is using a sample size of 100 employees to know with 95% confidence that the percentage is between 77% and 79%. The second scenario gives us very acceptable reliability for about one-fifth the cost.

Decision makers need to become comfortable defining the level of integrity they need their measures to have. And it shouldn’t—indeed can’t—ever be 100%. No-one can afford that.

Measures that misinform and mislead

One of the easiest assumptions to make when we attempt to measure performance is that once we have named the measure, everyone will automatically define it the same way we do. It was obvious to us, so it should be obvious to everyone else. But the truth is that there are usually several different ways to calculate any specific measure. Take the measure of On-time Delivery for example. What defines ‘on-time’? A freight company measured this with unwavering commitment for many years because this was one of the most important performance attributes for its customers. But even when the measure was tracking well, customers were still complaining about late deliveries. Why did the measure contradict the customers’ feedback? Well, in this case, it depends on how ‘on-time’ is defined, and by whom. The freight company defined it as the arrival of the truck to the destination terminal. The customers defined it as when they could actually get access to their freight from the destination terminal. And these two times can be days apart, particularly over a long weekend: if the truck arrives after business hours on Friday, the terminal gates don’t open again for customer access until Tuesday morning. The trend of on-time arrival of the truck at the terminal can be vastly different from the trend of on-time delivery into the customers’ hands.

A client of mine enjoys telling the following story about his experience with bringing performance measures to life. Peter was the manager of strategic sourcing in his company. When ‘cost rationalisation’ became a corporate priority, he went about designing a procurement strategy that would save the organisation as much money as possible without sacrificing the reliability or availability of the services and materials the organisation needed to do its core work. He also designed and developed a set of fantastic performance measures to track his new procurement strategy.

One of the measures was Procurement Overhead, which was the cost of running the procurement activities for the company—different from the cost of the services and materials procured. One of the major overhead costs was managing stock for inventory, so Peter was keen to get as much inventory ordered automatically through a stock management system in combination with long-term purchasing arrangements from suppliers to bring inventory levels down. He asked the inventory manager, Simon, to start measuring the percentage of inventory on auto-order and to set his target at 80%.

Time passed, and there came a day when Simon proudly announced to Peter that he had achieved the target of 80% of inventory on auto-order. To Simon’s dismay, Peter didn’t seem impressed. Instead, he looked confused and somewhat sceptical. Peter hadn’t noticed the reduction in Procurement Overhead he had expected to see when the auto-order system was established. Most of the time and effort in manually ordering inventory should have disappeared. Where was the correlation between Simon’s achievement of the 80% target and the costs? A thought occurred to Peter, and he asked Simon how he was calculating the percentage of inventory on auto-order.

Simon replied: by adding up the value of items that the inventory system automatically orders, and dividing that by the value of all the items held in inventory. Simon was adding up the dollar value, not the number of items. So, having managed to get a handful of multi-million dollar inventory items on auto-order—which represented only a fraction of a percent of the total number of items held in inventory—Simon felt he had achieved the 80% target. But those items were not costly to order because they were ordered infrequently and in small quantities. Simon’s behaviour was not geared toward reducing overhead cost at all. Peter needed at least 80% of the number of items in inventory to be on auto-order—the superfluity of cables, nuts, bolts, screws, nails and other items that were ordered regularly and in vast quantities. The problem was that Peter had not clearly defined the details of the measure he wanted Simon to track and improve. Simon automatically assumed what Peter was after, and neither realised this erroneous assumption until the data didn’t match up.

Peter was one of the lucky decision makers who quickly realised his performance measure wasn’t making sense to the person charged with implementing it. But it doesn’t always work out that way. Many decision makers don’t ever realise they are being misled by their performance measures because of unexamined assumptions about how those measures should be calculated.

Too many conflicting versions of the truth

One of the greatest sources of waste in the process of bringing performance measures to life is when the same measure is produced by several different people, and each measure tells a different story. This was the case for Martin, who was receiving something like 12 different versions of a measure of the cycle time of coal trains from a range of business analysts throughout his department. And because no two of these 12 different versions matched, he had no idea which one was the accurate measure of cycle time. Martin had 12 measures, but no information.

The measures of cycle time didn’t match for a few reasons. Some had to do with where the business analysts were obtaining the data they used to compute their versions of cycle time. Some reasons were to do with which data they chose (e.g. how they defined the start and end times of the trains’ cycles). Other reasons were to do with the calculation method for cycle time (e.g. whether to weight the values by train tonnage). It was an uncoordinated mess. Martin is the manager I referred to earlier in the book, who joked with me about wanting to put all the different reports he got (more than 40!) into a wheelbarrow and wheel it into a management meeting to impress upon his executive colleagues just how dire the reporting situation was.

But it’s no joke that so many organisations suffer this same level of chaos and waste in their reporting processes. It takes weeks to get to the bottom of messes like Martin’s 12 cycle-time measures, and even longer to shift the culture to prevent such problems from recurring. This is a form of waste we should not tolerate.

Reacting to invalid signals and missing important signals in the data

In the electricity sector, one of the common performance measures used to track the reliability of the electricity network—the system of cables, transformers and so forth that delivers the power to homes and businesses—is the System Average Interruption Duration Index, or SAIDI. In simplified terms, SAIDI represents the amount of time the power is off during an outage. The lower the values of SAIDI (i.e. the shorter the time the power is off) the more reliable the network is considered to be. A typical way to report SAIDI is using a graph with a 12-month moving or rolling average.³¹ Moving averages are often used to smooth out the seasonal fluctuations we see in a lot of financial data. With these seasonal fluctuations smoothed out, we are supposed to be able to see the overall trend in the data. For the manager who needs to track a lot of performance results this sounds heaven sent, which is why rolling averages are such a popular trend-analysis technique. But popular doesn’t mean appropriate.

FIGURE 12. SAIDI moving average
FIGURE 12. SAIDI moving average

The graph shown in Figure 12 is a measure of SAIDI for one of my clients in the energy sector. Managers concluded from this graph that reliability had rapidly and steadily improved over the period from January 1999 through to February 2000 when it levelled off and then started to improve again from October 2000.

But is this graph really saying what the managers interpreted it to mean? Remember, it’s a moving average. The value for May 1999 is the average of the previous 12 months’ results (from June 1998 to May 1999). So that downward trend our eyes see is not describing what’s happening in the period from May 1999 to February 2000. It’s describing what has happened on average since June 1998. Is this really the best description of how reliability is performing over time?

Take a look at the next graph in Figure 13, which is based on exactly the same reliability data but shows the actual monthly totals instead of the rolling 12-month average of these values. (This chart type is called an XmR chart.)

This alternative way of representing performance data tells us some very different things about reliability performance than the rolling-average graph did. It doesn’t show any gradual improvement (downward trend) at all, which is what the rolling-average graph suggested. This graph does, however, show us that in April 1999 there was a sudden downward shift in the overall level of reliability (a shift in the right direction for a measure like SAIDI), and an associated narrowing of the natural variability in reliability. Nothing in this chart suggests that reliability is gradually getting better like the previous graph did.

FIGURE 13. Actual SAIDI values without the moving average
FIGURE 13. Actual SAIDI values without the moving average

Two very different interpretations of exactly the same data, from two different methods of analysing that same data. Why did this happen? Remember that moving averages were designed to smooth out seasonal fluctuations to reveal the overall trend. Is that an appropriate analysis method for this data that has no seasonal pattern? Is the overall trend the only thing we need to know to manage reliability performance? The answer is no, on both counts, as the second graph makes clear.

When we use inappropriate analysis methods for our performance measures, we miss signals that are there waiting to be noticed, or we respond to signals that aren’t there but are just artefacts of the analysis method.

The sudden decrease in SAIDI in April 1999 was due to a change in the way SAIDI was calculated. (There was no industry-wide standard with utilities using slightly different methods until April 1999.) Performance hadn’t changed at all. If they had used Figure 13, which shows the actual monthly data, the management team would have seen the sudden decrease in April 1999 and looked into the cause before deciding what (if anything) to do.

No one is accountable or responsible

‘We’re all responsible for each of the strategic KPIs’, said a CEO when I asked which executive would be the performance owner for one of the new performance measures we had just designed for the company’s strategic direction. He hadn’t heard that when everyone is responsible, no one is.

Being responsible for a performance measure does not (and should not) mean you have to present your bum for a kicking whenever the measure heads in the wrong direction. But responsibility has to lie with someone, or the measure will be rendered ineffectual. Someone has to take the lead, monitor the measure, be the first to speak out when a signal appears, and initiate action if action is needed. This is the person who rallies the troops and makes sure all is being done to pursue and achieve the target when the measure shows things are heading off course. If the person responsible for a performance measure is doing all of this, their bum should be well protected from the CEO’s shiny shoe. This person has a very important job to do, and needs all the support they can get to do that job well.³²

Measures don’t drive performance improvement

Staff Turnover is a measure held dear by just about every HR Department. Yet just about every discussion I’ve had with HR professionals about this measure has centred on the same fundamental question: ‘How should we respond to it?’

The problem is that, unlike the electricity system reliability measure discussed earlier Staff Turnover doesn’t have a clean and consistent interpretation of ‘up is bad, and down is good’. Turnover measures employee retention, but is often used also a proxy for other goals such as staff engagement, organisational culture and employee satisfaction. Depending on the result we want, we’d interpret Staff Turnover differently. One of my clients had a very low Staff Turnover level, and wasn’t sure if they should be setting a target to reduce it further. After some discussion about how they felt about their current staff retention, performance and culture, it became clear this organisation really needed to bring in some freshly enthusiastic people, and help some of the long-tenured people who weren’t aligned with the organisation’s changing priorities to move on. Aiming for a lower Staff Turnover would work directly in opposition to what this organisation needed.

Another client organisation was very keen on reducing Staff Turnover because they wanted to retain their best people. If they kept their eye only on the overall Staff Turnover measure, they could reduce it by applying broad-brush staff retention strategies. But with that approach they’d be retaining the wrong staff and wasting their efforts unless they focused very specifically on the turnover of the employees they defined as their ‘best people’.

Decision makers need to think carefully about performance measures that aren’t really measures of performance, especially legacy measures that have been measured routinely for so long their original purpose may be long forgotten.

How do we avoid waste in the performance measurement process?

Bringing performance measures to life requires labour—extracting data from systems, collecting data that aren’t currently available, analysing the data to create the performance measure values, and carefully graphing and presenting the measure so it can announce relevant signals clearly. This labour is the discipline of conducting activities such as:

Designing surveys or operational data collection processes to collect performance measure data that have sufficient integrity

Extracting data from the various database systems that the organisation keeps, to make those data quickly and easily available for analysis and measure calculation

Preparing the data for analysis by removing outliers, correcting data entry errors and anomalies, and calculating ratios or extra data fields needed for the measure values

Performing the appropriate analysis and calculations to produce performance measure values and insights from the prepared data

Graphing the performance measure values to aid valid and quick interpretation of any signals the measures have to reveal

Extracting, collating, organising, cleansing, analysing and graphing additional relevant data to explain any signals revealed by the measures

Generating dashboards, scorecards or performance reports for suites of related measures, and additional commentary for interpreting and explaining the implications of the results

Disseminating or presenting the dashboards, scorecards or performance reports to their audiences to be used in their decision processes

Discussing performance signals, results and cause analyses to decide on appropriate courses of action to improve performance when improvement is required.

This labour is the performance reporting process. It happens manually through the efforts of researchers, information technology people, statisticians and business analysts, at least until a business intelligence system can automate much of this process (if doing so is worth the cost). But these activities of the reporting process will generate all the problems we’ve discussed above if there is no thorough definition of the performance measure.

Defining a performance measure means fleshing out the specifics of its calculation, presentation, interpretation and ownership. PuMP breaks down the process of defining a performance measure into eight steps:

Step 5.1: Description

Step 5.2: Intent

Step 5.3: Where it fits

Step 5.4: Calculation

Step 5.5: Presentation

Step 5.6: Response

Step 5.7: Owners

Step 5.8: Notes

Fleshing out these details specifies how to bring a performance measure to life, and takes the ambiguity out of sourcing, analysing and reporting the data, as well as using the data to answer questions about the performance result the data tracks. The PuMP Defining Measures technique saves us from wasting time trying to figure out which data to source, sourcing the wrong data, performing the wrong analysis, or misinterpreting the measure and subsequently making a wrong decision.

Having a clear framework for defining each of our performance measures in unambiguous detail goes a long way to solving these problems that cause so much waste and angst in many performance reporting processes. Defining performance measures takes some time, but the benefits are worth it: consistency in the meaning and calculation of each performance measure as it is used across the organisation, which is particularly important when we want to be able to compare performance among groups, divisions or processes. It’s also easier to communicate a well-defined performance measure to others so that its purpose and use are consistently understood. Updates or changes to each performance measure (e.g. when data sources or owners change) will be easier to control. Business analysts and intelligence officers don’t have to remember or guess how to put the measures together each time they need to be reported. And each performance measure’s purpose, interpretation and use is spelled out in no uncertain terms, making it easier to cull measures that aren’t worth the effort or are no longer relevant to the organisation’s strategy or mission.

None of the eight steps to define a performance measure is difficult. Let’s take some time to walk through each step and see how it’s done.

The 8 steps to define a performance measure

Step 5.1: Description

We need to give names to our performance measures. If we’ve done the PuMP Designing Measures step well, we’ll have the name already. We’ll also have a first draft of our measure’s description. The intent of describing the meaning of our performance measures is so that those who will implement or use each measure can quickly understand its meaning and purpose. The definition is a statement that clearly, simply and tangibly summarises the calculation procedure and the type of data on which the measure is based.

To write a great description for a performance measure, we ask the following questions:

What people, events or things does the measure monitor (e.g. employees, customers, invoices, policies, workplace accidents)?

What characteristics of these people, events, or things are we measuring (e.g. tenure, profitability, accuracy, uptake, frequency)?

How are these characteristics going to be quantified (e.g. an average, a percentage, a ratio, a total)?

What are the scope boundaries of the measure (e.g. full-time permanent employees, active customers, invoices for amounts greater than $1,000)?

If we include the answers to these four questions in our performance measure description, we can be pretty confident the measure’s meaning will be apparent to most people, providing we:

Use a sentence to describe what we want to monitor (as opposed to bullet points or phrases)

Use sensory-specific language

Word it using plain English so almost anyone can understand it (i.e. avoid jargon and complex or ambiguous language).

Here’s an example of a description for a measure called Overall Customer Satisfaction: ‘The quarterly average rating of how satisfied customers are overall with our service, for customers who purchased from us within the quarter.’ Customers are the people being monitored. Their rating of overall satisfaction with our service is the characteristic of these people that we’re measuring. We’re using an average to do the measuring. And we’re interested only in those customers who have bought something from us during the measurement period.

Let’s work through a slightly adapted example from one of my clients, a freight business. One of the freight management division’s goals is to ‘Deliver our customers’ consignments perfectly’, and one of their performance measures for this is Consignment Service Failure Rate. A clear, complete description for this measure might be as shown in Table 9.

TABLE 9. Description of Consignment Service Failure Rate performance measure
TABLE 9. Description of Consignment Service Failure Rate performance measure

Step 5.2: Intent

The next step in defining our performance measure is to clearly express why the measure is needed. This is a check to make sure we have a real need for the measure. Sometimes we think a measure is a good idea, but when we think about it in the light of a new day (the day we define it), its worth can look less convincing. Even for the measures we are convinced are worthwhile, writing the intent for that measure can help convince others.

The intent statement is simply our answer to questions such as:

Why is it so important that we measure this?

What’s the driving priority behind this measure?

What important difference will this measure help us make?

For the performance measure example of Consignment Service Failure Rate, Table 10 shows how they might express the intent of this measure:

TABLE 10. Intent Consignment Service Failure Rate performance measure
TABLE 10. Intent of Consignment Service Failure Rate performance measure

Step 5.3: Where it fits

Performance measures usually live in hierarchies or systems of measures, which means that any measure will have at least one relationship to other measures (e.g. cause-effect, companion, conflict, or lead-lag). A measure can also link to a particular business goal, objective, strategy, business unit, process or activity. When we are managing dozens or even hundreds of performance measures, how we organise them for easy sorting, grouping and filtering is of undeniable importance. That’s why it’s useful to include a ‘where it fits’ section in our performance measure definition template.

There are several ways to classify or categorise each of our performance measures for convenient organisation, including:

Its relationships to other performance measures

Which result from our Results Map it monitors

The level of the Results Map where it belongs

Which strategic or strategic theme, key result area, critical success factor, goal or objective it monitors

Which business process it relates to

Which functional unit or department it belongs to

It’s not essential that we include all these suggestions in our definitions of measures. And we might come up with some other ways to classify measures. But the key is to give considered thought to what we include, so we can manage where each measure fits in our organisation’s entire framework of performance measures.

The freight management process performance measure of Consignment Service Failure Rate might be classified and organised in the business’s performance measurement system as shown in Table 11.

Step 5.4: Calculation

Step 5.4 in defining each of our performance measures is to write down the formula for analysing raw data to produce the values for the measure. We need to define the calculation frequency, clarify the scope, and identify the required data items and their sources.

TABLE 11. Where the Consignment Service Failure Rate performance measure fits
TABLE 11. Where the Consignment Service Failure Rate performance measure fits

There are many ways of summarising or aggregating data so we can get some picture of overall performance. For example, we can use averages, medians, frequency counts, ranges, percentiles, ratios, or simple totals. Unless the summary statistic for each measure is clearly and fully defined, the measure might be calculated in different ways or different timeframes by different people, making comparisons through time or with targets or other measures completely meaningless. Choosing the wrong summary statistic can also mean the performance measure doesn’t truly represent the thing about which we are making our decision, so people will be confused about what the measure really means and will likely interpret the measure’s values wrongly or inappropriately. Or they’ll give up and stop using such a confusing piece of information.

Performance measures don’t come out of thin air. It’s easy to take for granted that we know where the data for them will come from, that it will be simple to get the data, and that calculating the values for the measure will be obvious. But these are very risky assumptions. Being super-specific when we define exactly how to calculate our performance measures plays a huge role in keeping our performance measurement system’s costs down and its utility high.

In essence, when we define how to calculate our performance measure, we will detail four specific aspects of the calculation: the formula, the frequency, the scope, and the data needed for the calculation.

THE FORMULA

The first element of our calculation is to clearly document the formula for our measure. This formula is used to transform the raw data into our performance measure values. For the Consignment Service Failure Rate measure, we can see from the description that it is the rate of service failure occurrences compared to the number of consignments. So the formula would simply be:

We’ll talk about what ‘Service Failure ID’ and ‘Consignment ID’ mean in a moment. How we write the formula matters. We need to be sure it matches the summary statistic alluded to in our performance measure’s description, and we need to make the required data items clear. We can write the formula in mathematical symbols, such as using E to depict sums, or the formula can be written in words, such as ‘the total number of service failures divided by the total number of consignments’. The above formula for Consignment Service Failure Rate is half mathematical symbols and half words. It doesn’t matter greatly which way we choose, so long as it is easily and consistently interpreted by those who will use it.

THE FREQUENCY

The second aspect of defining a measure’s calculation is the frequency, or how often we will do the measuring. That depends on a number of factors. To work out the ideal frequency, we’ll trade off several things. We need to look at the measure frequently enough to pick up important signals of change and have enough time to respond before things get out of hand, but also infrequently enough to avoid getting lost in too much noisy detail. The frequency of calculation should line up as practically as possible with reporting processes that are already in place, such as weekly team meetings or monthly reports (assuming these reporting structures are appropriate). But this has to be balanced with the availability of data so that enough data will be generated in the time frame to make reporting worthwhile.

For our freight management division’s Consignment Service Failure Rate measure, a weekly frequency is appropriate because the team meets weekly to review performance, and there are many consignments and (unfortunately) many service failures each week.

THE SCOPE

A third aspect of defining a measure’s calculation is its scope: what will be included or excluded from the measure. It helps, when defining the scope, to be very specific about the population from which the data for the measure will come. A population is a collection or list of the people, events or things we are monitoring with our chosen measure. To define a measure’s population, we can start by asking ourselves the following questions:

What are the units or things, or who are the people, the measure is describing? Are they deliveries, freight damages, customers, sales, workplace accidents, etc?

Where can these units, things or people be found? On delivery trucks, visiting our stores, in the sales system, in the warehouse, etc?

When or over what time frames will these units, things or people be measured? Every time they occur, during September each year, at the end of each day, during the night shift, etc?

For example, the freight management division needs to define the scope of two things in its measure of Consignment Service Failure Rate: consignments and service failures. Every consignment is clearly identifiable using a consignment number (Consignment ID), and all consignments are in the scope of this measure. The division has also clearly defined the scope of service failures, so everyone knows which problems are relevant to include in the calculation of Consignment Service Failure Rate:

Misdirection (a consignment sent to the wrong destination)

Damage (a consignment that is damaged in any way)

Loss (a consignment declared lost)

Late arrival (a consignment not arriving on time at the receiving customer)

Surplus (a consignment for which we can’t find who the receiving customer is)

Shortage (a consignment that did not arrive at the receiving customer in full)

In many situations, we might be interested in measuring parts or segments of our population as well as the whole. This is called segmentation or classification, and is a very simple way to start drilling down into our performance measures as part of our analysis to understand their signals or trends. In the case of the Consignment Service Failure Rate, the division might want to look specifically at each type of failure.

When we want to segment, we first decide which characteristics of our population units make up the criteria for segmentation (e.g. age, revenue, location, gender, weight, freight type, destination). We use these characteristics to construct our segmentation criteria. The next step is to assign each population unit to the appropriate segment—in our example, specific service failures to one of the service failure types. Then, when we calculate the values of our performance measure, we calculate the values overall as well as for each of our segments. For example, the freight management division could calculate their Consignment Service Failure Rate for each of the six service failure types listed earlier, or only for the service failure types that happen most frequently.

Defining the scope of our performance measures is especially important when using sampling as the means of gathering the data (as opposed to gathering data from every single population unit). If our sample is not representative of the target population of our performance measure, then the data we collect from that sample will lack sufficient integrity. A natural counterpart to ensuring our performance measures are representative of the population about which we are making decisions is that when we make those decisions, the conclusions we draw apply only to that population and no-one or nothing else.

THE DATA

The fourth aspect of defining a measure’s calculation is to detail the data items—the observations or records—of the characteristics our measure will quantify. Often we find those data items already exist in our business database systems. There are times, however, when we don’t have the data we need for a new performance measure. If that’s the case, we need to give detailed thought to the type of data required for the measure so we can either find an appropriate source outside of our business or develop an internal data collection system for these data.

Clearly detailing the data a measure requires means doing enough that it’s easy for someone to quickly locate the right data for the measure. The clues about the data on which our measure depends lie in the formula for the calculation. If we’ve expressed the formula well, we’ll see that it is made up of specific data items combined into a mathematical or statistical expression that will produce the values of the measure. We will often need to define or identify several data items to give us the complete set of data we need to construct our measure. Some measures can also share data items, which is an important point to note when we want to economise and standardise data collection in our organisation. And it’s why having a data referencing model is so important. A data referencing model is a documented framework of all the data collected and stored in our organisations, and how those data are defined, formatted, organised, linked and managed.

It doesn’t hurt to add the source for each data item if a source already exists. This source might be a formal organisational database or data warehouse, a manual data collection process such as a check-sheet approach to logging faults or defects, or a computer-based data repository such as Microsoft Access or Excel.

We don’t need to be pedantic about defining our data items. We simply need to have enough information to know what data to look for and where to go to find them. For example, in the formula for Consignment Service Failure Rate, there are two obvious data items: ‘Service Failure ID’ and ‘Consignment ID’. There are also two data items that are not so obvious. ‘Consignment Delivery Date’ is one because Consignment Service Failure Rate is being calculated weekly, so we want to count only the consignments that were delivered in each specific week. And ‘Service Failure Type’ is needed to make sure that only the services failures that match the types defined in the scope portion of the measure’s definition are counted in the formula.

Table 12 shows how the full calculation of the freight management division’s measure Consignment Service Failure Rate might be defined.

TABLE 12. Calculation of Consignment Service Failure Rate performance measure

Step 5.5: Presentation

At Step 5.5 in defining each of our performance measures, we’ll choose the best means of displaying our performance measure values so that they tell us what we need to know.

Presentation is everything. When it comes to the crunch, if we have presented something (no matter how informative, relevant, or important it is) in a way that doesn’t get the audience’s attention or help the data tell their story, then we have achieved nothing. It’s important to choose and standardise the best way to visually display our measure, so its signals are clear, valid, and not distorted or misrepresented in any way.

And when we have a standard presentation method for each measure, its familiarity makes it easier for users who will continue to monitor the measures over time. With the presentation method clearly defined, we can provide a guide to those reporting the measure, so they properly balance visual usefulness with visual interest.

DEFINING THE COMPARISON TYPE APPROPRIATE TO THE MEASURE

Comparison is the basis of meaning, and no single value of a performance measure is going to tell us anything unless we compare it to something else, such as a target, a standard, or historic values. Almost every question can be answered by comparing something with another thing. Unless we are very clear about the comparisons we need to make to answer our driving questions, we might experience any or all of the following:

We misinterpret the measure, or use it out of context.

We see a signal in the measure that actually isn’t there.

We don’t see a signal that really is there.

We have an answer to a question that isn’t the question we designed the measure to answer.

For performance measurement specifically, we’ll likely choose from a few basic comparison types:

Trend-over-time comparisons are the most useful of all comparison types for understanding the true level of current performance when we include enough historic data, and are measuring frequently enough to have sufficient measure values for a trend analysis.

Actual-with-desired comparisons are like comparing the current level of actual customer satisfaction with a targeted level or industry best-practice benchmark.

Point-with-point comparisons are like comparing the values of a measure taken too infrequently for true trend analysis, such as employee satisfaction based on an annual culture survey.

We can choose more than one type of comparison. For example, we can choose a trend-over-time comparison accompanied by a comparison with target or standard. Or we can choose several comparisons of the same type, such as a trend-over-time comparison for our measure as a whole and also for each segment (e.g. customer group or incident type).

CHOOSING THE BEST GRAPH TO DISPLAY THE MEASURE’S VALUES

Once we know which comparison type will best draw the insights out of our performance measure, we’ll make a deliberate decision about the visual method to display and communicate those insights. This means choosing a graph type that suits our comparison type. We want to choose the presentation method or graph type that will display the values produced by the formula we defined for our measure and highlight the comparisons we defined. Does that sound obvious? I am stating the obvious simply because of how many times I’ve seen people define their measure as, let’s say, a weekly average to be presented as a trend over time, but then graph it as a monthly cumulative sum in a bar chart.

Table 13 offers suggestions for which graph type to choose based on the type of comparison we want to highlight.

Yes, there are lots of other methods for analysing and presenting data. But when our focus is on monitoring the performance progress over time, the comparisons and graph types in Table 13 are the most suitable. Of course, we will often present other types of data analysis along with our performance measures for cause analysis and so on. We’ll delve deeper into that in the next chapter.

The presentation of the freight management division’s Consignment Service Failure Rate measure is defined as shown in Table 14.

Step 5.6: Response

Step 5.6 in defining our performance measures is to describe how users should respond to each measure. We think about the signals that the measure should be capable of showing us (such as upward trend or shift, downward trend or shift, or no change), how each signal should be interpreted, and what action should be taken in response.

TABLE 13. Appropriate presentation methods for the different comparison types
TABLE 13. Appropriate presentation methods for the different comparison types

TABLE 14. Presentation of Consignment Service Failure Rate performance measure
TABLE 14. Presentation of Consignment Service Failure Rate performance measure

The ultimate outcome of performance measurement is improvement. Action is needed to produce improvement, and action is not the same as motion. There’s a difference between doing something and doing something that brings a desired result. There are many ways to improve performance, especially when choices are based on quick and dirty methods such as gut feelings about what should be changed, traditional moves such as downsizing or cutting budgets or reorganising, or whatever is easiest to fix. But the point of performance measurement is to choose actions that give the biggest return. If we highlight what these actions could be and how we would know if the time was right to take those actions, we’re helping our performance measures lead to the desired outcome of performance improvement.

Our performance measures are there to give us advice about what is going on in our business so we can choose the most appropriate way to manage its performance. This means we will choose different types of actions depending on what kind of advice a measure is giving us or what kind of change in performance it’s signalling. The basic signals are:

Signal 1: Performance has reached or exceeded the target.

Signal 2: Performance is improving at a rate fast enough that the target will likely be met.

Signal 3: Performance is improving, but not fast enough to reach the target.

Signal 4: Performance is stable and not changing.

Signal 5: Performance is getting worse.

Signal 6: Performance is unpredictable or chaotic.

Let’s look briefly at each of these signals and consider the actions we might take in response to those signals. We’ll delve more deeply into interpretating and responding to performance measures in Chapter 10.

SIGNAL 1: PERFORMANCE HAS REACHED OR EXCEEDED TARGET

The best signal to get from our performance measures is that the target has been achieved. As with the signal of performance tracking confidently toward its target, this is a great sign our strategies and improvement projects have worked. But as we have seen in previous examples, such signals are not a sure sign because of extraneous factors outside our control. So when performance measures show us the target has been achieved, we check a few things before we celebrate:

Check performance measures that have a causal relationship to this measure, testing whether achievement of targets in those measures is a likely reason for achievement of the target for this one.

If we don’t find any reasonable explanations for achieving the target in related mesaures, we check the external environment for extraneous factors that might be the cause, and perform some deeper exploratory analysis to find out the root reason that the target was met.

Once we’re sure that the target has definitely been achieved and we know why, we take some further steps:

Standardise any successful changes brought about by our strategies and improvement projects that resulted in this target being met, and make these changes the new ‘business as usual’.

Unless there is another target that has been set for this measure (as in the case of staged targets), perhaps it’s time to move the performance measure off the strategic or operational scorecard and refer to it only if it signals deteriorating performance in the future.

Celebrate the success, and reward the people who made the improvement happen.

Contrary to what we might think, exceeding a target is not always better than meeting a target. Exceeding a target can be a potential waste of resources and time that might have been better spent fixing other, more important performance shortfalls.

So consider responding to this signal as follows:

If any strategies and improvement projects aimed at achieving the target are still in progress, critically assess whether they should be scaled back or stopped, and whether the time and resources going toward those projects should be invested in addressing another important performance shortfall.

SIGNAL 2: PERFORMANCE IS IMPROVING AT A RATE FAST ENOUGH THAT THE TARGET WILL LIKELY BE MET

This is a good signal. The fact our measure is indeed tracking confidently toward its target level of performance is a great sign that our strategies and improvement projects aimed at achieving the target are working. But such signals are not a sure sign because, just as extraneous factors can cause deterioration in performance beyond our control, they can also cause an improvement in performance beyond our control. (The economy is a good example in both cases.) So in this case we do the same first three steps as for Signal 1:

Check performance measures that have a causal relationship to this measure, looking for any trends or shifts in those measures that might explain the successful improvement in this one.

If we don’t find any reasonable explanations for the improvement in performance in related measures, we check the external environment for extraneous factors that might be the cause, and perform some deeper exploratory analysis to find out the root reason that performance is improving.

Standardise any successful changes brought about by our strategies and improvement projects that resulted in the improvement, and make them the new ‘business as usual’.

We continue to execute our strategies in pursuit of reaching the target.

SIGNAL 3: PERFORMANCE IS IMPROVING, BUT NOT FAST ENOUGH TO REACH THE TARGET

All improvement in performance is good when it’s planned. But sometimes improvement is not big enough or fast enough to reach the planned targets. If the targets still matter, and our measure is signalling the improvement isn’t happening fast enough to reach the target in time, we need to intervene:

Check performance measures that have a causal relationship to this measure, looking for any slower-than-planned improvements in those measures that might explain the slower-than-planned improvement in this one.

Perform some deeper exploratory analysis to double-check the reason performance is improving but not fast enough, and compare that to what was expected in the improvement plan.

If improvement of this process is a priority, either check the improvement project to re-evaluate its apparently ineffective action or reconsider if the target is too big a stretch for the resources available to pursue it.

The real value of targets is not in achieving them, but in pursuing them. They pull us in the direction of improvement, and as long as we are improving performance, reaching a specific target may not matter.

SIGNAL 4: PERFORMANCE IS STABLE AND NOT CHANGING

Performance values will always vary to some extent, and variation does not necessarily mean change. If our performance measure values are varying consistently within the same band or range, and we don’t see any values breaking away from this consistent but random pattern over time, the measure is signalling that nothing is changing. Sometimes that’s good, but not if we expect to see improvement. So in this case we do the following:

Check performance measures that have a causal relationship to this measure, looking for any lack of trends or shifts in those measures that might explain the lack of improvement in this one.

Perform some deeper exploratory analysis to find the key variables that can be influenced to get performance to start improving.

If there is an improvement project in place for this measure, check its focus and progress to re-evaluate its apparently ineffective action.

If there are no projects in place to make this improvement, establish an improvement project to manage some of the key controllable variables that affect performance.

Of course the assumption is that if we are measuring performance it’s because we want to improve performance. If we have a performance measure and we don’t care if it doesn’t change, then it’s not really a performance measure.

SIGNAL 5: PERFORMANCE IS GETTING WORSE

Depending on our performance measure, worsening performance might be evidenced by an upward trend or shift as in the case of Expenditure or Rework Hours, or by a downward trend or shift as in the case of Customer Satisfaction or Profit. If we can pinpoint when the performance began worsening, we have a better chance of finding out why and taking successful action to turn the trend around:

Check performance measures that have a causal relationship to this measure, looking for any trends or shifts in those measures that might explain the result for this one.

Perform some deeper exploratory analysis to find out the root cause of performance worsening, and assess options for controlling that root cause.

If we haven’t already started to address the root cause, focus an improvement project on doing so.

If we have started to address the cause and should be seeing some improvement by now, check the improvement project to re-evaluate its apparently ineffective action.

The last thing we want to do with this signal is find someone or something to blame. This is a moment of truth where the strength of our belief that performance measures are our teachers is put to the test.

SIGNAL 6: PERFORMANCE IS UNPREDICTABLE OR CHAOTIC

When performance is unpredictable, it fluctuates wildly from week to week or month to month. This chaotic behaviour is symptomatic of a business process that is out of control because the process steps or the inputs used lack standardisation. In this case, we don’t try to improve performance. We get the process under control first:

Check performance measures that have a causal relationship to this measure, looking for any patterns in those measures that might explain the chaos in this one.

Perform some deeper exploratory analysis to find out what factors are causing performance to spin out of control (e.g. a lack of uniform procedures, or constant readjustment or ‘correction’ in how steps are performed or how equipment is set up—also known as tampering).

Establish improvement projects to remove or control the factors causing performance to be chaotic (e.g. put in a quality system, create a flowchart showing how the process should be performed, and then train workers to follow a consistent process).

We cannot improve what we cannot control or influence.

These six signals are the ideal range of signals to which we want to be ready to respond quickly and with a clear course of action. But we can start out with a simpler array of signals until, as Chapter 10 will explain, we get more comfortable with how to validly interpret performance measures from a strong statistical foundation. Here are three simple signals we can embed in the “response” section of our performance measure definition:

Performance is improving (there is a shift or trend in a desirable direction).

Performance is unchanging (there is no shift or trend at all).

Performance is worsening (there is a shift or trend in an undesirable direction).

For example, the freight management process performance measure Consignment Service Failure Rate could have the response definitions shown in Table 15.

Step 5.7: Owners

Step 5.7 in defining our performance measures is to nominate who is responsible for monitoring each measure, sharing its signals with others, and initiating actions for improving it. This person is the performance owner for the measure. We might also find it useful to nominate a definition owner and data owner.

PERFORMANCE OWNER

The best and most meaningful performance measure in the world can be rendered utterly useless if no-one is clear about who should interpret and use it. Likewise, too many cooks spoil the broth. If everyone tries to take control over improving the same area of performance, then time, effort and resources will almost certainly be wasted. The most appropriate person to be the performance owner of a particular measure is the person responsible for managing the process, function or activity the measure is monitoring. That person has access to the data, knowledge of how that area works and, most importantly, is in a position to make decisions about how to improve that area. The performance owner of a measure is responsible for:

Monitoring (looking at) the measure over time

Interpreting its trends and patterns and seeking causes for them

Communicating this information to people affected by that performance area

Initiating action to improve performance in that area

Following up to be sure that actions are having the desired effect on performance.

TABLE 15. Responses to Consignment Service Failure Rate trends
TABLE 15. Responses to Consignment Service Failure Rate trends

The performance owner is not the person whose bum gets kicked if the performance measure goes in the wrong direction or the target is missed. No one person can be completely responsible for the actual quantitative or qualitative values of a measure because its values are almost always based on the interplay of many factors, most of which are outside the control of any one person. These are factors like organisational policies and procedures and the actions of other people, who are human and imperfect. So focusing blame on one person is focusing attention away from the real causes—and the real opportunities—for improving performance. What’s the use of holding the call centre manager responsible for a measure of Average Handling Time (the average time it takes to handle customer enquiries) when that time is largely dependent on the quality of products and services other departments in the company are delivering? Collaboration fixes problems, not blame.

Giving ownership of a particular performance measure to someone does not exclude others from taking a role in monitoring, interpreting or improving the performance it tracks. The performance owner is simply the person who has the best experience, skill and knowledge to lead the interpretation of and response to the measure. So we always make doubly sure that the owner of a performance measure has the knowledge, experience, skill and authority to monitor, interpret and respond to his measure.

DEFINITION OWNER

A performance measure’s definition owner is the person who knows the best way to define the measure’s calculation and enough understanding of its role to also confidently define the other aspects of the measure, including its intent, description, presentation, and the appropriate responses to the trends that it reveals. The definition owner will coordinate the creation of the measure, and maintain the definition over time so it reflects the needs of the performance owner and the organisation.

Ideally, the definition owner will be independent of the area of performance the measure tracks to ensure the definition remains focused on organisational needs and is not changed to suit the needs of any individual. We don’t want the performance owner of an on-time delivery measure who feels threatened by that measure changing the definition of on-time from ‘within 1 hour of promised time’ to ‘within 1 day of promised time’.

DATA OWNER

A performance measure’s data owner is the person who either owns the sources of data required for the measure, or knows the right ways to access or extract it. This person plays a big part in bringing a measure to life as quickly, easily and accurately as possible.

The data owner for a measure should be the person to whom we’d turn for finding the right data, writing queries to filter and extract the data from databases, checking and correcting any data quality problems, or running simple summary statistics for the measure. Often this person is a business analyst or IT officer. Sometimes they are the same person as the definition owner.

But a word of caution: don’t depend on the data owner to work their magic and produce the values of a performance measure in a puff of smoke. Part of their role ought to be to document the data filtering, extraction, cleaning and summary procedures so the organisation does not lose these procedures if that person leaves. One of my clients had a very smart gentleman in such a role. He created and managed dozens of spreadsheets containing complex layouts of data calculations that were the sole basis for many of the organisation’s performance measures. If he was sick, the performance measures had to wait. No-one else knew how the spreadsheets worked, or even where they were stored in the organisation’s system. Even more frightening was that this gentleman was a year or two away from retirement.

One of the best places to document the data filtering, extraction, cleaning and summary procedures is in the performance measure definition itself, in the data items part of the calculation section. For example, we could state the names of data extraction queries in the data source column in the calculation section of the Measure Definition template.

Continuing the example of our freight management division’s performance measure of Consignment Service Failure Rate, the owners could be defined as shown in Table 16.

TABLE 16. Owners of Consignment Service Failure Rate performance measure
TABLE 16. Owners of Consignment Service Failure Rate performance measure

The data items part of the calculation section in the measure’s definition could also be updated as shown in Table 17 to include more specific instructions for sourcing the data.

TABLE 17. Data Items for Consignment Service Failure Rate performance measure, with sourcing queries
TABLE 17. Data Items for Consignment Service Failure Rate performance measure, with sourcing queries

Step 5.8: Notes

Notes include improvement ideas, cautions, intended life span, analysis and reporting procedures, and other points to consider when reporting or using a measure.

There is no such thing as a perfect performance measure. The usefulness of any measure is governed by factors such as its definition, who uses it, how it is being used, and what it is being used for. Because all of these factors can change over time, any performance measure is useful only in a specific context. Giving that measure the flexibility to stay useful in an environment of continuous change is an important part of defining it.

TABLE 18. Notes for Consignment Service Failure Rate performance measure
TABLE 18. Notes for Consignment Service Failure Rate performance measure

Including notes in our performance measure definitions is a great way to document anything that relates to the construction, interpretation or use of the measure that we don’t want to formalise as a regular section in the performance measure definition template. Notes that we might want to keep track of for our performance measures include:

How data for the measure are collected, such as documented data collection procedures

How data are turned into information, such as documented data analysis procedures or online systems

Ideas for future improvement of the measure’s definition and usage

Current limitations on the use or application of the measure

Its original source (if, for example, we got the measure from another organisation or an industry standard).

The freight management division’s performance measure of Consignment Service Failure Rate might have a few notes as shown in Table 18.

The full definition of the Service Failures measure

Table 19 brings together all the elements of a performance measure definition for the freight management process measure of Consignment Service Failure Rate.

TABLE 19. Full definition for Consignment Service Failure Rate performance measure

We can modify and streamline this definition framework (as long as we test it) by giving it to the people who will be responsible for implementing it (the definition owner and data owner). If they don’t understand it, or they don’t implement the measure the way we had intended, the definition needs to be improved.

Here are some elements of the framework for defining measures that users have suggested and employed over the years:

The expected or intended lifetime/lifespan of the performance measure, which is usually a date by which the measure will no longer be required or applicable.

The date the measure was created or added to the measure dictionary.

The measure’s current status, such as ‘draft’, ‘in review’, ‘signed off’, ‘active’ or ‘retired’. These types of features include whatever makes sense for us to keep track of where our measures are in their implementation progress.

The measure’s past, current and future targets. My preference is usually that targets belong in business plans, but keeping a history of targets for a measure can allow for interesting reflection.

A history of changes made to the definition of the measure, such as changes to the data sources, formula or owners.

How the performance measure is reported, such as documented reporting procedures, online systems, or the names of reports in which it features.

Which direction in the measure’s trend represents improvement. For example, for Rework Hours, going down would be the direction of improvement whereas, for Customer Satisfaction, going up would be the direction of improvement.

The Corporate Performance Measure Definition Dictionary

A very large part of the waste and effort in performance measurement activities is the lack of coordinated management of an organisation’s collection of measures. Even if everyone is using the same performance measure definition template, effort and waste can result from duplicating measures, losing measure definition documents, or losing track of the latest updates. We need a central repository where all performance measure definitions are kept, maintained, and periodically reviewed for consistency, completeness and relevance. I call this a Corporate Performance Measure Dictionary.

The Corporate Performance Measure Dictionary can be part of the quality system, document control system, or knowledge management system (if the organisation has one). The organisation’s performance management office, strategy and performance team, or whoever owns its performance measurement system is responsible for keeping and updating the entire set of performance measures for the organisation.

Take inspiration from one of the world’s largest and most-used performance measurement collections, KPI Library.³³ KPI Library has a KPI librarian, whose role is to make sure all key performance indicators are listed once (and only once) and have a single definition. In an interview, KPI Library’s founder Karel van der Poel explains this is the most important and challenging job for the KPI librarian.

‘If I were setting up a performance measure framework or project in a big organization, I would definitely have the same role in the organization, because we’ve seen with many large organizations that people re-invent the wheel, there are spread sheets everywhere and people are measuring the same things, but naming the measures slightly differently, measuring them at a different frequency. It’s just a waste of effort, a waste of time, and it makes it really hard to look at performance’.

The best way to make sure our Measure Definitions are good enough is to test them out. Before the Measures Team gets together again for Step 6 in the PuMP Blueprint, give them a week or two to use their Measure Definitions to start collating the data for a few of their new measures. Have those data ready in a spreadsheet for the PuMP Pilot Step 6 meeting, where you’ll help them use that data to produce a performance report that rocks.

Take Action: Build your performance measure definition dictionary

To begin with, we might prefer to define a few measures using a simple Microsoft Word-based template. But we need to make sure we don’t end up with a mountain of uncontrolled Word documents floating all over the organisation. As soon as we can, it’s best move to a database method of logging and managing performance measure definitions. I have both a Microsoft Word template to help us get started, and a Microsoft Access database we can use or model as our corporate performance measure definition dictionaries.

Go to the book’s website (www.pumpblueprintbook.com) and download the PuMP Measure Definition template.

Chapter summary

We can only define a well-designed measure, not a vague concept of a measure. The PuMP Designing Measures technique comes before the PuMP Defining Measures technique. As I mentioned earlier, I once attended a training course where the instructor asked us to define a measure of innovation using a template similar to the PuMP Defining Measures technique. We spent almost all the time allotted for this activity figuring out what the measure of innovation should be, and didn’t get to fill in any of the definition template.

The PuMP Defining Measures technique fleshes out the necessary detail to implement our performance measures so they come to life as we intended. These details include:

a clear description

a reminder of each measure’s intent

where each measure fits in the organisation’s suite of measures

how it should be calculated and from what data

the best way to present the measure

the signals to look for and how to respond to them

and who the owners of the measure and its definition are.

Manage a suite of performance measures using a corporate Performance Measure Definition Dictionary. Make sure someone fills a role similar to that of a librarian, managing and caring for the complete collection of performance measures used throughout the organisation or business.

Before the Measures Team starts Step 6 in the PuMP Blueprint, you might like to give them an extra few days or a week to source some data for their new measures. But don’t let a lack of data delay Step 6 any more than a week or two. Get the data collection ball rolling, and keep moving through the Blueprint.

9

Step 6: Reporting Performance Measures

‘Computers speed the process of information handling, but they don’t tell us what the information means or how to communicate its meaning to decision makers. These skills are not intuitive; they rely largely on analysis and presentation skills that must be learned.’

— Stephen Few, www.perceptualedge.com

If our performance reports are stacking up in a pile, unread and unused, then they’re obviously not ‘stacking up’ well as sources of invaluable insight to guide performance improvement.

It’s an emotional thing, performance reporting. Executives give up the precious little time they have for their families and nine holes of golf to instead paw through piles of strategic reports often more than an inch thick. Or they leave the piles of reports on their desk and make decisions from their gut instead. Managers earnestly trawl through operational reports to check whether anything needs a bit more positive light thrown on it. Supervisors and teams scoff cynically about the volumes of time and effort they waste reporting tables of statistics that track their daily activities to audiences they never see or hear from.

What makes performance reporting such an upsetting ordeal?

Almost all performance reports and dashboards suck

Managers have little time, a multitude of responsibilities, and a lot of performance issues to manage. They need performance information presented in a way that helps them do their jobs more easily, not in a way that consumes their time for the pointless satisfaction of bureaucracy. High-quality performance reports—those that contain meaningful performance measures and analysis—are exactly what busy managers need to confidently give their time and attention to what truly matters most and not to anything else. So performance reports need to provide the content that truly matters most, in a form that is fast and easy for managers to digest. But most performance reports are just the opposite.

They are thrown together in an ad hoc way, with little thought behind the structure and organisation, making it very hard to navigate to the information most relevant or urgent for keeping strategy execution on track.

They are cluttered and cumbersome, containing too much detail that drowns out the important signals and distracts with trivial and non-actionable information.

Their information is displayed in indigestible tables, silly graphs, and dials and gauges that aim to entertain but unintentionally result in dangerous misinterpretation.

The layout is messy and unprofessional, wasting visual real estate, detracting from the report’s importance, and disengaging readers before they find the insights they can use.

They are time-consuming to produce, with most of the time going into sourcing, cleaning, and summarising data, rather than analysis and gleaning insights.

Just about every problem with a performance report, or performance dashboard for that matter, falls into one of these five areas. Grab one of your performance reports, and take a look at it as we discuss the problems it may very well have.

Ad hoc and hard to navigate

If a report has no obvious structure—for example, it lacks clear and comprehensible section headings—then our speed and ease in navigating the report will usually be directly proportional to the number of times we have looked through the same report in the past. We have to read a poorly structured report often before we work out its logic (if we ever do), and to learn how best to find what we need within its pages.

It’s easy to overlook incomplete information when a report has no logical or deliberately chosen structure. We just take what’s in it for granted without reflecting on whether it contains the right information, or whether the information is presented in the best order. When a report lacks a structure that links directly to strategic, tactical or operational objectives, we also have difficulty linking the information to decisions. We’ll often see reports like these stacked neatly on the boardroom table next to each decision maker, but they will rarely be opened up at the most appropriate time to use them.

There are many ways to structure a performance report. But the structure should be a practical decision, not a creative one. Great performance reports and dashboards virtually act as the agenda for performance review meetings, guiding the priorities, cause analysis and dialogue about how to respond to performance results. Performance reports and dashboards are companions to decision making. So a report or dashboard can’t be structured sensibly until we first understand the decision process that it exists to inform.

Cluttered and cumbersome with too much detail

It’s almost a natural law of performance management that reports know only how to grow, not how to condense. Even though we all know that no-one reads large reports, we still produce them—and struggle to find the information we need in them. I hear story after story about how several years ago Bob the Manager wanted a particular measure—say Revenue Earned per Employee—and it’s been reported ever since even though Bob wanted it only for a two-month-long project and he left the organisation two years ago. After a succession of Bobs add their ad hoc measures, performance reports become bloated and unusable.

Many of our organisations suffer from the belief that ‘we just might need that information one day, so let’s keep it in there just in case’. This is a symptom of not focusing or prioritising, and of not really knowing what the purpose or role of the performance report is. With that rationale, the performance report is trying to be all things to all people for all time. When attention scatters, nothing matters. (That should be your new mantra.) Performance reports are supposed to focus us, so we can take action on what most needs action. We can’t solve every problem, set targets for every aspect of our work, and monitor everything that can be measured. The processing power of our conscious brains, coupled with the number of hours in a day and the limitation of resources we always complain about, forbid us from doing this. I’ll say it again: Performance reports are supposed to focus us on the things that matter most.

The bulk of many performance reports also makes navigation through them a nightmare, and producing them becomes an unbearable effort. The volume of data that must be collated and analysed keeps analysts punching away at their computers through their lunch breaks and late into the night at the end of each month or reporting period. And, at the end, users of the report don’t really know how to distinguish which information should get priority, and which is noise drowning out the important business performance signals.

Performance report writing tends to be an evolutionary process. It can shift, change and adjust almost every month. Often, many people who contribute content to the report such as graphs, commentary or project progress updates (and usually way too much of the latter two), don’t really know how the information will be used. Most believe the information won’t be used at all. So there is a lot of variation in what ends up in the report, how much ends up in the report, and what gets emphasised.

We need to be very deliberate and disciplined about the types of information that performance reports should contain. And that requires a rationale and some guidelines.

Tables and graphs cause misinterpretation

Tables are the traditional way to present performance information, mostly because financial reports like the profit-and-loss statement have always been tabular. But tables often cause us to miss important patterns or trends. They usually show only current performance compared to last month’s, and that kind of comparison is absolutely incapable of telling us the full story of where performance has been and is headed. Tables also require us to read and hold a lot of numbers in our heads, rather than quickly showing us the most insightful and revealing patterns as visual methods like graphs can do.

Tabular performance data takes a long time to read and interpret, and different people can draw different conclusions from the data. Also, tables encourage us to focus on details of specific individual results, not on the bigger picture of relationships (such as cause-effect or lead-lag) among a collection of results. Tables severely constrain our view of performance data, and the speed and accuracy with which we can extract its insights. The insights come from the patterns in the data, not the data points themselves.

Graphs can take up less space than tables, are far more interesting to look at, convey more information much faster, and give us more control over the way people will interpret the data presented. Rather than using a table of ‘this month to last month’, ‘this month to the same month last year’ or ‘year-to-date to target’ comparisons, we can use line charts to display a performance measure’s historic data. This will quickly reveal patterns of change, such as trends and sudden shifts in performance over time. Tables simply can’t do this. With all the research on visual cognition demonstrating that graphs are more effective for conveying quick insights than tables are, it’s not wise to pander to the personal preferences of a vocal minority. The majority of people will make reliable and faster interpretations about what performance is really doing by looking at graphs rather than tables.

In my workshops, I show the numbers laid out in Figure 14 and ask participants to describe the signal, observing how long it takes until someone sees it, or until everyone gives up. (And so far everyone has given up.)

Many seconds pass, and the silence in the room proves the point. No-one can see a signal in this table of numbers. Can you? If you do see a signal, how long did it take you to find it? There is a signal in this set of numbers. If these numbers were the monthly values of a performance measure, we’d certainly want to know about this signal. The same numbers are presented in Figure 15, and I’ll bet you can see the signal there.

Figure 14
FIGURE 14. Can you see the signal in this table?

The signal is a sudden increase in the overall average level of performance, about one-third of the way into the time series. That’s definitely a signal we want to know about—and know about quickly. But the table in Figure 14 doesn’t show it to us. Even if we were tracking this-month-to-last-month comparisons, we wouldn’t realise this shift had happened. We can see it only when the larger trend is plotted. We’ll delve more deeply into the topic of seeing signals in our measures in Chapter 10, but for now I think the value of graphs for quickly revealing insights is clear.

As far as creating graphs goes, Microsoft Excel is certainly an easy enough graphing tool to use. We can whip up all manner of creative and colourful graphs in seconds. However, we need to remember that the software can’t decide what the best type of graph is for our data. Excel doesn’t know or understand the question that the graph should answer. Unfortunately, many performance reports are filled with Excel’s default graphs (or someone’s idea of a good-looking graph), which encourage invalid interpretation of the information. And invalid interpretation leads to bad decisions.

Figure 15
FIGURE 15. Do you see the signal now?

Now let’s open up a real can of worms about visual display of measures: dials and gauges. Puhleeze! About the only argument I have found in support of dials and gauges that are supposed to make a report look like the dashboard of a car or cockpit is that they look groovy. However, their groovy appearance does not make up for their inability to display valid signals in the underlying performance data, the difficulty of reading them, the amount of dashboard real estate they take up relative to the amount of insight they provide, and the risks of gross misinterpretation they pose. Performance reports and dashboards are for informing, not entertaining.

All that distracting ‘chartjunk’, as visual information expert Edward Tufte³⁴ calls it—legends, data labels, grid lines, fancy symbols, gaudy colours, three-dimensional bars and cones, and linear trends lines that explain less than 10% of what’s going on in the data—turn decision making into a circus. We struggle to adjust as graph types change on every page, to pinpoint true signals in the data amidst the entertaining chartjunk, and we get headaches from the visual assault of lurid colours and multiple measures presented on a single chart.

We need to adopt proven methods for designing graphical displays of our performance measures. We need to ensure we validly and quickly interpret what the measures are saying rather than jumping to the wrong conclusions and taking the wrong actions—or taking no action at all because we lack the stamina to make sense of what is presented to us.

Messy and unprofessional

Performance reports put together with two font styles—one for the normal text and one for the headings—aren’t just boring and monotonous to the eye. They are also hard to navigate, so they frustrate our attempts to rapidly find needed information.

In an age where word processing and spreadsheet software is easy to use, and offers a plethora of formatting tools (colour, typefaces, font sizes, borders, shading, and so on), there is no excuse for not adding a layer of visual information that aids us in navigating and interpreting a performance report. This layer of information could be called ‘visual prioritisation’—highlighting the most important results to make it easy for us to prioritise what we read first and to know how to interpret the information.

Dull layout and design deaden our interest, cause us to put the report on the bottom of our reading pile, and make us feel uncomfortable or awkward using the report. It’s not too hard to learn the basics of good visual design. I learned from Robin Williams (not the actor), who wrote The Non-Designer’s Design Book.³⁵ Williams discusses four very basic principles of visual design: proximity, alignment, repetition, and contrast.

In performance reports, proximity is about where information is placed in the report and the use of white space to visually ‘chunk’ related information together, such as collating related measures under strategic goal headings. Alignment is about how space on each page or dashboard screen is used so graphs, explanations, cause analysis and recommendations have obvious connections to one another. Repetition is about consistency, such as using the same graph type and formatting to display performance measures for every trend-over-time comparison. And contrast is about making the important things really stand out, such as signals of sudden or gradual change in performance, or the attainment of a target.

Professional visual design achieves a few things. First, it gives the report a personality or mood that invites us to read it and feel confident about understanding and interpreting it. Second, professional visual design can make navigation and orientation through the report easy, as alignment, white space and formatted headings visually separate different groups or types of information. Third, clever formatting (using colour, typefaces and symbols) can make the most important information in the report almost jump off the page. The time required to read and interpret the report ends up being a fraction of what it might otherwise be, and the process is many times more enjoyable and satisfying, Stephen Few³⁶ is a modern-day master of visual design for performance information, and you simply must follow his good work for both the sensible practices of visual quantitative analysis he describes and the rationale for those practices. He truly is my mentor when it comes to displaying performance measures usefully and useably in any medium, dashboard or otherwise.

Time-consuming to produce

It can take weeks to pull together a monthly performance report. You need to:

extract data from organisational database systems

clean the data to remove outliers and correct obvious errors

organise and massage the data to get them into a format appropriate to perform calculations or specific analyses

update graphs and tables

chase people for cause analysis, comments, or project updates

collate everything into a single document

wait for managers to review the draft version before finally publishing and distributing the final version three weeks after the end of the month.

What an ordeal!

Much of the performance reporting process is ad hoc, unresourced, and dependent on the free time of busy people who don’t see how contributing to the report is part of their ‘real work’. There can be a lot of rework time in chasing people who ought to know their contribution is needed each month, extracting, organising and analysing data, and collating information together into a single document and making it as presentable as possible.

Most of the reporting process can and should be automated. Business analysts responsible for producing performance reports each month can (and should) spend most of their time using their analysis skills to dig for insights in the data and adding context and cause analysis to the performance measures. Extracting and analysing the data, and presenting the performance measures themselves, can be almost entirely automated.

Reports and dashboards should slap us in the face (in a good way)

Putting a really good performance report or dashboard together isn’t an ad hoc process. It’s quite systematic, and once we’ve created templates for our organisation’s performance reports and dashboards (based on the PuMP principles and expert advice from the likes of Stephen Few) we will enjoy some very worthwhile benefits:

A simple template that makes it easy to plug in the updated numbers and commentary for each report, thereby reducing production time and effort.

A professional look and feel that engages decision makers and gives the report instant credibility.

Layout that makes the report quick and easy to use in decision making, so it actually gets used.

Content that feeds decisions and directs our decisions, virtually forming the agenda for performance review meetings.

Dialogue that focuses on performance results and solutions for performance improvement rather than debates over data integrity or measure relevance.

Creating performance reports and dashboards that rock isn’t guesswork. There is a basic process for working out how we should present our performance measures. This process stems from an understanding of how our audience is most likely going to try to give meaning to the measures. In fact, if we present our measures really well, our audience shouldn’t have to try at all. They will automatically understand what the information in front of them means, and how they should respond to it.

There are five steps for designing a performance report that is compelling, easy to navigate, and profoundly useful in decision making to improve organisational performance and strategy execution. It’s the PuMP Designing Performance Reports technique.

Step 6.1: Structure to strategy

Step 6.2: Answer what, why, and now what?

Step 6.3: Use graphs that signal

Step 6.4: Design to engage

Step 6.5: Automate, automate, automate

So, with a copy of one of your organisation’s performance reports or dashboard screen shots in hand, let’s see how it stacks up as we follow an example of building a performance report using the five steps of the PuMP technique for designing performance reports.

The 5 steps for designing reports and dashboards that rock

Step 6.1: Structure to strategy

The first design element for a performance report is to structure the report in alignment with the organisation’s strategy, which means the results that matter most. Structuring to strategy means we choose the headings in a performance report consciously to be consistent with business plan goals, objectives or priorities. With this structure, the report will hang together logically so each layer of headings forms a natural grouping. For example, the major headings might reflect business goals, while subheadings within each of those major headings might focus on the impacts of specific initiatives chosen to reach each of those goals or on the individual performance measures for those goals. The table of contents constructed from these major headings and related subheadings will allow decision makers to navigate the report at a glance and choose to read what is of direct relevance to their areas of responsibility or interest.

Figure 16 shows an excerpt from the quarterly report of the electrical services unit at the Oregon Health & Sciences University (OHSU). This is a paper-based report because the team was new to performance measurement and not ready to invest in a dashboard application. They used what they had (Microsoft Excel and Word) because it was more important that they start tracking critical results than waiting for the perfect reporting platform.

The electrical services unit’s operational strategy comprised several key result areas, which formed the major headings in the unit’s performance report. The subheadings each related to a performance measure, and were clustered under the key result areas to which they related. The structure is clear and succinct, and the addition of the smiley-face symbols (a form of ‘traffic light’ symbol, which we will discuss a little later on) means the table of contents gives a quick snapshot of the quarter’s performance. The electrical services unit (or any other user of this report) can quickly identify the priority information they should look at first. In this case, it’s likely to be the On-time per SLA, Estimate accuracy, and Backlog hours measures because the unsmiling symbols for these measures indicate that performance is not where the team wants it to be.

Figure 16
FIGURE 16. Table of contents from OHSU electrical services’ quarterly report

You might not like the smiley-face approach that this team chose for highlighting measures that need priority attention. But it’s not about the choice of symbol. It’s about making it easy for the report users to prioritise quickly and easily.

Step 6.2: Answer what, why, and now what?

The second element to address in designing performance reports is the content: the performance information that the report should include. A great guideline for choosing content is to focus it on answering the questions ‘what?’, ‘why?’, and ‘now what?’ They are shorthand for:

1. What is performance actually doing, relative to what we want it to do?

2. Why is it doing that?

3. Now what are we going to do about it, to move it closer to where we want it?

These questions focus us on monitoring how well we are achieving our goals, and how likely we are to achieve the targets associated with our goals. Providing only the information necessary to answer these questions will ensure the report is as simple and concise as it can be, and that it focuses on highlighting the insights that will improve strategy execution and goal attainment.

When we answer the question ‘what?’, we’ve got to make sure the graph we choose to display the performance measure encourages valid interpretation of that measure. The graph needs to be very clear, and not open to debate about the signal the measure is showing. Is performance improving, getting worse, or staying the same? Is performance improving fast enough to reach its target on time or not? We’ve got to be clear about which signals we need to be able to detect, and we need to know the best way to highlight those signals when and if they happen. Chapter 10 will give us a specific tool that answers the question ‘what?’ in the best possible way.

The next question is ‘why?’ This question is about making sure the report encourages the decision makers to inquire and learn about how to improve performance, rather than blame someone or something for how performance is behaving. If the report shows only a downward trend in the performance measure of Revenue, it’s too easy for decision makers to say, ‘It’s the economy’ or ‘Our industry’s struggling all over right now. That’s why this measure’s not looking good’ or ‘We’ll have to put the wind up the sales manager again’. That’s not the way to approach performance management. Instead, decision makers should be asking ‘Why are we seeing this signal? What are the real reasons? What are the factors that are within our circle of influence and have a bearing on this performance result?’ Answers to these questions are the clues to what kind of action to take, regardless of what is happening in our industry or in the economy, or how we feel about the sales manager. Cause analysis is the beginning of answering these ‘why?’ questions.

The third question, ‘now what?’, means exploring and deciding on the best way to respond to the performance measure’s signal to help performance reach its target or head in the direction we want it to go. Performance reports can help stimulate the discussion about how to respond to a performance measure by providing recommendations aimed at removing or controlling the root causes of a signal. Reports can even review potential solutions that have already been pilot-tested and recommend the solution that had the best impact on performance relative to the solution’s cost.

That’s the content of the performance report: the information that adds the right flesh to the bones of the report’s strategy-aligned structure.

OHSU’s electrical services unit adopted the same layout for every page of their quarterly report. Each page was devoted to one performance measure, and each page answers all three content questions with sections entitled ‘Results’, ‘Interpretation’ and ‘Response’. Figure 17 shows an example page from that report.

Step 6.3: Use graphs that signal

The third design element essential to producing effective performance reports is using graphs that accurately and quickly convey the signals showing what performance is truly doing right now. The three basic signals answer the questions ‘Is performance improving?’, ‘Is it getting worse?’, and ‘Is it staying the same?’ However, as we discussed in Chapter 8, when we compare actual performance to a target we end up with six signals. Here’s a reminder of those signals:

Signal 1: Performance has reached or exceeded target.

Signal 2: Performance is improving at a rate fast enough that the target will likely be met.

Signal 3: Performance is improving, but not fast enough to reach the target.

Signal 4: Performance is stable and not changing.

Signal 5: Performance is getting worse.

Signal 6: Performance is unpredictable or chaotic.

Each graph we put in a performance report should focus on only one performance measure. We don’t try to put multiple measures in the same graph. Instead, we put enough information in each graph to help that one measure tell its story. Gridlines aren’t needed, and we don’t need legends either if we’re displaying just one performance measure at a time using consistent formatting and symbols to indicate targets, the measure’s time series, and so on. We should not be afraid to use the same chart type for each performance measure. We don’t have to try to make the report ‘interesting’ by changing the chart type. In fact, we’ll be much better off keeping things as consistent as we can so users build a familiarity with the way the report looks and feels. They won’t waste attention on figuring out the changing graph types from page to page. Instead they’ll turn their attention to the changing signals, which is the important thing for them to look at. In Chapter 10 we’ll learn about a very powerful chart type I recommend as the best way to visually present all of our performance measures.

Figure 17
FIGURE 17. Typical content layout from OHSU electrical services’ quarterly report

When we include graphs in the ‘why?’ section of the performance report for cause analysis, we’ll use not only line charts but other types of charts as well. A Pareto chart (a bar chart in which the bars are ordered by size) would be a common choice. Imagine a bar chart of counts of various types of customer complaints. If we order those bars by the frequency of those complaints, from largest to smallest, we have a Pareto chart. Look at the longest bar, and we’d know straight away that it represents the biggest problem our customers have. And it is the biggest thing we need to do something about if we’re going to improve performance and reduce the overall number of complaints.

Many types of analyses and graphs can help us investigate causes of signals in our performance measures. I won’t go into detail about these other types because others in the field of visual analysis and analytics do a superlative job at helping us understand the best methods of visual analysis and visual design. My go-to reference collection for designing graphs in performance reports and dashboards includes Stephen Few’s Show Me the Numbers,³⁷ Information Dashboard Design,³⁸ and Now You See It.³⁹

Suffice to say, we need to make every graph in a performance report as clear and concise as we can. These five basic principles will help us do that:

1. Answer the question

2. Highlight comparisons

3. Optimise simplicity

4. Maintain integrity

5. Be visual.

Let’s look at each principle in detail.

ANSWER THE QUESTION

Answering the question we want a particular performance measure to address is fundamentally about identifying why we want the graph, and then making sure the graph has enough anchors to link its information back to the driving question. There can be far-reaching consequences to using a graph that doesn’t clearly relate to the question we want to answer. We can end up doing what doesn’t need to be done, not doing something that does need to be done, or doing it the wrong way. Here are the most important things to think about to ensure a graph answers the right question for the measure it illustrates:

Choose relevant and appropriate data for the graph.

Near the graph, mention the meaning of the measure(s) that it illustrates. (This meaning can be the description from our Measure Definition—see Chapter 8.)

Explain the scope of interpretation of the measure (from the scope section in the Measure Definition).

Provide the source that produced the measure and its credibility. (Link to the Measure Definition, or reference the source of data.)

Keep each graph focused on answering just one question.

If we pay attention to specific parts of a graph’s anatomy, we have power over how well it answers our question. Main titles, subtitles, footnotes and axis labels should make the purpose and scope of the graph clear and unambiguous. The data on which we construct the graph should directly answer the question the graph is designed to inform. Simply by looking at the graph itself, we should be able to easily tell what question it was designed to answer.

Contrast how well the graphs in Figure 18 and Figure 19 make clear what question they are supposed to answer.

Figure 18
FIGURE 18. This graph fails to make clear the question it is answering.

Figure 19
FIGURE 19. This graph contains enough information for us to know what question it answers.

HIGHLIGHT COMPARISONS

Highlighting comparisons is important because nothing has absolute meaning on its own. Knowing that Average Overall Customer Satisfaction for our highest priority customers is 7.5 isn’t necessarily enough information to understand how well we are performing. Their satisfaction has meaning only when we compare it to the satisfaction level we wanted to reach, the satisfaction they’ve had over previous years, or the satisfaction they have with our competitors.

TABLE 20. ‘Ready reckoner’ for choosing the best graph
Table 20

Comparisons give meaning to things. Knowing which comparison we want to highlight is key to creating useful graphs. Each graph type has its own special way of highlighting comparisons in data, and we can quickly choose the best graph for a particular application using the ‘ready reckoner’ in Table 20. The only popular chart type I leave out is the pie chart. Pie charts are useless for so many reasons. They lack a common visual baseline for easy and valid comparisons, they require us to compare two-dimensional shapes where a one-dimensional symbol (like a bar) shows a value in a way we can perceive much more accurately, and they take up more space than necessary. A bar chart will always work better whenever we think we want a pie chart.

Figure 20
FIGURE 20. A bar chart is a poor way to highlight a trend-over-time comparison

If we make careful choices about the graph type we use (e.g. bar chart versus line chart), we have power over the graph’s ability to highlight the right comparison and answer the right question. See how the line chart Figure 21 makes it much easier to read the trend over time in Number of Complaints Received than the bar chart of Figure 20?

OPTIMISE SIMPLICITY

The conscious mind of a human being can process only around seven chunks of information at any one time. So if we want our graphs to be useful to human beings, we have to aim for no more than seven pieces of information (including titles, data, axis labels, and so on) in each. Here are some tips on how to simplify graphs to reduce the types of information that the viewer has to process.

Use tick marks only when interpretation of the data depends on them

Avoid gridlines

If gridlines are unavoidable, use very fine or light-coloured lines

Keep titles, legends and supplementary information relevant and succinct

If the goal is to convey actual numerical values, use a table (rather than a graph that has data labels above points or bars)

Present only one measure in a graph

If more than one measure has to be included (e.g. a second measure has to be shown for context), make the primary measure visually dominant perhaps by using a bright or saturated colour, and the secondary measure visually subtle perhaps by using a lighter colour

Keep the audience’s attention focused on comparisons in the data, not on entertaining artwork.

Figure 21
FIGURE 21. A line chart is one of the best ways to highlight a trend-overtime comparison

We exert the utmost power over the simplicity of our graphs when we are:

working out how much data to present

deciding how to use and format borders, titles, footnotes, legends, labels, axes and gridlines

choosing the size, colour and types of symbols to represent the data (e.g. shapes, bars, lines, etc.)

You can see many of these features cluttering the graph in Figure 22, and the simple solution in Figure 23.

Figure 22
FIGURE 22. Gridlines, lots of axis tick marks and too many measures on a single graph make it too complex

Figure 23
FIGURE 23. Using small multiples with consistent and simple formatting optimises simplicity

MAINTAIN INTEGRITY

Once we have confidence in our data and information, then we need to have confidence in how we use and present them. The following techniques will give our graphs maximum integrity.

When using bar charts and Pareto charts, always start the vertical axis at the logical minimum of the data range. Otherwise, comparisons between the bar heights will be misleading.

Axes must be continuous and consistently scaled. Don’t take chunks out of the middle of them in an attempt to condense the graph area.

Flag when data is presented as cumulative sums and moving averages (rather than as individual and independent values) because this greatly impacts our interpretation of patterns and trends.

When using regression lines, trend lines and forecasts, show measures of their statistical reliability. For example, include the measure of R² with a trend line. The R² measure indicates how much variation in the data is explained by the trend line, and therefore how useful the trend line is in explaining patterns in the measure. (We’ll learn more about this in Chapter 10).

Near the graph, report the reliability of the data, such as its source and any known deficiencies in data completeness or accuracy.

Avoid using two-dimensional shapes (such as the slices in a pie chart) and three-dimensional shapes (such as the 3-D effect on bars in a bar chart) to portray one-dimensional data, because 2-D and 3-D shapes distort comparisons.

We exercise power over the integrity of our graphs by giving thought to the quality of the data available, and carefully choosing the graph and symbol types, the axis ranges and amount of supplementary information we include to guide interpretation of the graph. Can you see the lie the first chart in Figure 24 is telling us? The truth is in the second chart.

BE VISUAL

Our brains process visual images very rapidly, often encoding a signal, message or conclusion before we even have time to reason about the process. Remembering that graphs are visual images helps us understand the importance of using their visual attributes to encode the information in them effectively. The following techniques are simple but go a long way toward enhancing the visual performance of our graphs.

Figure 24
FIGURE 24. A non-zero baseline distorts the size of comparisons, failing to maintain integrity

Be selective in the shadings and patterns used in the graphs so that they are easy on the audience’s eyes.

Aim for common visual baselines so differences and similarities in the data are easy to detect.

Be consistent in graph layout and formatting when regularly producing the same or similar graphs.

Use a consistent alignment (i.e. left, centred, or right) throughout a graph.

Use white space to separate the title(s), legend, data points and the graph itself from other text.

Use contrasting shades and colours to make comparisons and other high-priority information stand out.

Repeat the same fonts and symbols to maintain consistency.

If the data has no natural order (i.e. the values are not ordinal), sort bars by size.

Let the data fill the data space by minimising the scale on the y-axis.

Be aware of when white space is meaningful—such as when the data is percentages and comparison to 100% is useful.

Minimise the size of the graph area as much as practical.

Use colour to make the graph easy to interpret.

We exercise power over the visual competency of our graphs by:

carefully choosing patterns and colours used as ‘fills’, as well as line types and symbols

taking care in the way we order or sort the data

using consistent elements in repeated graphs of the same measure.

Contrast Figure 25 with Figure 26 and notice how much more visually graceful the latter is. The clumsy target line in Figure 25 is represented as a point above its due date in Figure 26, and it is given the same colour as the central line to visually emphasise that this is the comparison the graph is designed to make.

In Figure 27, we can see that the OHSU electrical services unit has chosen very simple graphs and uses line charts to display their Utilization performance measure over time so it’s easier to pick up signals. (Utilization is the percentage of time charged to work orders.)

Figure 25
FIGURE 25. Too much white space, unnecessary use of colour, and using lines for targets lowers the visual competency of a graph

Figure 26
FIGURE 26. Simple and consistent standards for colours, symbols and targets helps a graph to be visually competent

The electrical services unit could make a couple of easy changes that would render this graph simpler and visually stronger. First, the graph contains a lot of white space. Because the most important comparison is actual performance relative to the min-max range, they could easily truncate the axis to start at around 50% and stop at around 90% without being misleading. Second, they could get away with not labelling every single week in the horizontal axis. If they marked perhaps every fourth week, they could use horizontal labels to make the graph look cleaner and simpler.

Figure 27
FIGURE 27. A line chart presents the performance measure Utilization in OHSU electrical services’ quarterly report

To explore the ‘why?’ question in their performance report, the electrical services unit used the Pareto chart shown in Figure 28. This chart was constructed to illustrate the breakdown of where time is going when it can’t be charged to work orders. Straight away we can see that paid time off is the biggest use of time that isn’t charged to work orders. Miscellaneous administrative work comes in second. These two factors together explain the majority of time that can’t be charged to work orders. This is a very simple and powerful visual presentation of a useful cause analysis for the Utilization measure.

Colour adds a dimension of information in this Pareto chart by showing the relative contributions of each team within the electrical services unit to the activities that cannot be allocated to work orders. Colour is often used like this, but in this case it’s not clear what question the coloured information is meant to answer.

Is the contribution of each team being shown because each team wants to see the breakdown of their own time on activities that can’t be charged to work orders? If so, a better way to answer this question would be to provide a Pareto chart for each team in addition to the overall chart. Because the overall Pareto chart doesn’t give a common visual baseline for each team, comparisons of the causes for a specific team are impossible to make quickly. And because each team has a different amount of time allocated to the various non-work-order activities, the order of teams in a separate Pareto chart of team contributions would be different than it is on this overall chart.

Figure 28
FIGURE 28. A Pareto chart used for cause analysis in OHSU electrical services’ quarterly report

Or is the question about which team process most needs reviewing to make the biggest increase in the overall time allocated to work orders? A better way to answer this question would be to use a bar chart or a Pareto chart that compares the total non-work-order hours by team. If the teams have varying numbers of members, a better measure for the bar chart would be total non-work-order hours per person for each of the teams.

This brief analysis of what the colours in the chart in Figure 28 do and don’t show us demonstrates why it is important to be clear about exactly what question we want to answer, what data will answer that question, and then to carefully choose each element of the visual presentation of the data that addresses the question.

Step 6.4: Design to engage

The fourth design element for performance reports is using formatting and layout to engage people. We can have a bit of fun with designing a report that fascinates and interests people. But design should not interfere with getting the right message across, quickly and validly. In his book The Visual Display of Quantitative Information, Edward Tufte introduces the ‘data-ink ratio’:

A large share of the ink on a graphic should present data-information, the ink changing as the data change. Data-ink is the non-erasable core of a graphic, the non-redundant ink arranged in response to variation in the numbers represented. Then,

Data-ink ratio = [data-ink] / [total ink used to print the graphic]

= proportion of a graphic’s ink devoted to the non-redundant display of data-information

= 1.0—proportion of a graphic that can be erased without loss of data-information

Our goal is to keep the data-ink ratio (or the data-pixel ratio) in mind as we design our entire report, not just the graphs within the report. We want to make the report easy to use, approachable, familiar and comfortable. We also want it to highlight the right signals, messages and answers. And we want to do all this with the smallest amount of ink or number of pixels we can. These principles inform our design of the layout and formatting of our performance reports.

LAYOUT SHOULD MAKE NAVIGATION FAST AND EASY

The layout of a report has a big bearing on how approachable it is. If we have a consistent layout, and consistent placement of each type of information on each page, then navigation is one less thing the users of the report need to worry about. Headings are always at the top of the page, footers contain the page number, and paragraphs and sections are separated with adequate white space. Users should know the layout is going to always be that way so their eyes automatically go to the right place every time they turn the page or click through to another screen. In the case of printed reports, the layout should also make the report easy to handle physically. For example, if users have to sift through a folder full of reports, then single-sided landscape may be better than double-sided portrait. But if the report is spiral-bound, double-sided printing can work.

FORMATTING SHOULD MAKE INTERPRETATION FAST AND EASY (AND ACCURATE)

The way we use formatting (text styles, colours and symbols) in a report can and should add meaning. We want to make major headings stand out from minor headings using font size, and to give headings bolder typefaces or a different font type than paragraph text. If a particular style of bullet is used to format the paragraph about causes of performance changes, then the same bullet style should be used for all paragraphs about causes of performance changes. The relative priority of performance results can be flagged visually to speed up interpretation by using colour or symbols. For example, signals of a decline in performance can stand out if we place a red cross next to the graph, while a big green tick might accompany a signal of improvement in performance. We don’t want to use fancy symbols or colours if it doesn’t add meaning or aid interpretation. And we don’t change the colour in each graph just to decorate or add visual interest. All we accomplish with these kinds of superfluous symbols and colours is we make things complex and confusing, so readers have to spend more time interpreting the information.

Adopting formatting standards reduces confusion in interpretation and use of performance reports. For example, I usually use the same colour for the line and symbols for a performance measure’s values. I always use a target point that’s a solid blue circle, sitting above the date on the horizontal axis corresponding to when that target should be achieved. So whenever we see that object in one of my charts, we know it’s a target. If we use the same typeface, font size and colour for the title of each chart, users won’t be distracted by variation. Doing things this way is not being difficult or fussy. It’s about deliberately using formatting practices we all know so they engage people, and making it easier and faster for readers to accurately interpret the performance information.

Look again at the table of contents of the quarterly report for OHSU’s electrical services team in Figure 16. Notice how they’ve used formatting with the very specific purpose of turning the table of contents into an executive summary (or snapshot) of overall performance? The smiley-face symbols effectively highlight which performance areas are tracking well and which need attention. Now, look again at Figure 17 and see how easy it is to navigate naturally through the three important questions (‘what?’, ‘why?’, and ‘now what?’) as your eye follows the headings down the page. Every page is formatted this way, so the entire quarterly report (which contains 21 performance measures clustered under six key result areas) is quick and easy to digest.

Step 6.5: Automate, automate, automate

The final design element for performance reports is not so much about the report itself but rather about how it’s produced. Specifically, when we design a performance report we should think about how much of it we can automate. Performance reports take a lot of time to produce: gathering the data, analysing the data, collating the data. The more of that we automate, the more of our time is freed to act on the information in the report and improve actual performance. Automation is possible, even with a manual process where Excel and Word are used to create performance reports. For example, we can automate queries that extract data from the database systems in the same way each week or month so the values are ready to add into the Excel spreadsheet. We can set up the Excel spreadsheet so that, once the new data are added, the graphs automatically update. And we can automate the performance report itself (the Word document) by using styles for common items such as headings and bullet points and body text, and hyperlinks to graphs in the Excel spreadsheets that update at the click of a mouse button. Then all we do to produce each issue of the performance report is:

1. Run the data extraction queries.

2. Add the data into the Excel spreadsheet.

3. Update the graph hyperlinks in the Word document.

4. Share the draft report with the measures’ performance owners so they can look at the signals, interpret them, and prepare any required cause analysis.

5. Do the cause analysis, and add a summary of the analysis to the performance report.

6. Update the recommendations or planned responses for the measures.

7. Publish the report.

Business analysts and others who put reports together should be spending the bulk of their time on steps 4 and 5 above. That’s their greatest contribution to performance improvement: analysing data to help decision makers see more clearly why they’re getting the results they’re getting, and what options they might have to do something about it. Business analysts and people doing reporting should not be spending most of their time formatting, creating graphs, extracting data, etc. So we do what we can over time to automate as much of the process as possible.

If we’re using dashboard applications, we’ll find automation is even easier. For my own company’s performance report I use Tableau⁴⁰ software. It reaches into my database systems, extracts the data, and once I have set up analyses and graphs in Tableau for my current set of performance measures I simply drag and drop each measure’s graph onto a dashboard. In essence, once my data are updated I just open Tableau and the report is refreshed and updated in seconds. There really are some simple and inexpensive ways to automate performance reporting, and produce reports quickly enough to deliver the required information about strategy execution and achievement before it’s old news.

Once your Measures Team has their performance report template set up and populated with the measures for which they have collated data, their focus should be on one thing before you meet for Step 7 in the PuMP Blueprint: collating data for a few more of the remaining performance measures, and arranging them in simple line charts. That starting point will help them when they meet for Step 7 to interpret signals in their performance measures.

Take Action: Design your performance report

Even though dashboards and business intelligence systems will likely be the best way to construct performance reports in the long term, I would suggest we start as simply as possible. That’s why the template I have for you to download uses Excel and Word to create a simple but powerful performance report.

Go to the book’s website (www.pumpblueprintbook.com) and download the PuMP Report Design templates.

Chapter summary

Too much time is wasted in most performance reporting processes in ad hoc efforts to extract data, analyse it, and collate it into reports. Time is also wasted when users of the reports can’t easily find insights they need from the information, and when they misinterpret the signals in the performance measures.

Good performance report design involves:

structuring the report so it aligns with the organisation’s goals or objectives

including only the information that can show what performance is doing, why it’s doing that, and how to respond

using graphs that simply, clearly and concisely highlight the real signals in the measures

professionally designing the layout and style of the report to engage users

automating as much of the reporting process as possible so analysts can focus on doing what they do best: analysing the data to reveal insights.

Create a standardised performance report template based on the PuMP principles for designing reports, and use that template consistently throughout the organisation so readers of the report can quickly, easily and accurately digest the performance information they need.

When your Measures Team’s new performance report is a good draft—not a perfect one—then move them on to Step 7 in the PuMP Blueprint. They are going to learn about a graph that will step their performance reports up a couple of notches on the power meter.

10

Step 7: Interpreting Signals from Measures

‘[A]utomobile sales are tallied and reported every 10 days. These values are always compared to the sales figures for the same 10-day period from the previous year. Do you remember how sales were doing during this 10-day period last year? Of course you don’t! So just what is the basis for making such a comparison? Why would this comparison provide additional insight into the current sales figures? It seems that comparisons of this sort are built on the implicit assumption that last year was normal. You do recall everyone talking about how normal last year was, don’t you?’

— Donald Wheeler, Understanding Variation

We’re at a critical junction in the performance measurement process. We face an obstacle so endemic, so menacing, so unconsciously ingrained in management culture that if we don’t surmount it, it could render our performance measures little more than decorative dashboard entertainment. It could even drive performance in completely the opposite direction from where we want it to go. And no, I’m not exaggerating.

The obstacle is our misinformed beliefs and bad habits in interpreting performance data.

Stop reacting to noise

The most common methods of interpreting performance data—of looking for signals of improvement or deterioration in performance over time—include:

Comparing this month’s measure value to the same month last year, or to budget, a target or a year-to-date value

Comparing the 12-month pattern in measure values of the current financial year with the 12-month patterns in previous financial years, generally in stacked line or bar charts

Comparing the slope of a linear trend line through measure values with the horizontal (equivalent to no change) to draw a conclusion about the overall direction of change

Comparing a moving or rolling average line (that smooths out the seasonal variation in measure values) with the horizontal to reveal the underlying direction of change.

These interpretation methods dangerously influence the decisions made by their users because if we apply all of these approaches to the same set of data, the conclusions drawn from each can be very different.

Let’s look at these common but flawed methods of interpreting performance data using an example from Tualatin Valley Water District (TVWD) in Oregon, USA. One of their operational performance measures is Fluoride Hours Above Maximum. The ideal result is to have a very low number of hours during which fluoride concentrations in the water are above the set maximum value. The data for this measure are collated in Table 21.

Now we’ll look at the fluoride measure using each of the four common methods of interpreting data.

THIS MONTH COMPARED TO THE PREVIOUS MONTH AND TO YEAR-TO-DATE

The table shown in Figure 29 is a very typical layout for a monthly performance report. Measures are listed down the left-hand side of a table, and several columns provide the values for the current month’s actual performance, the previous month (or the same month last year), the year-to-date average, and % difference calculations for all of the comparisons.

TABLE 21. Monthly values for the measure Flouride Hours Above Maximum
TABLE 21. Monthly values for the measure

FIGURE 29. Comparing this month
FIGURE 29. Comparing this month’s Fluoride Hours Above Maximum with last month’s and year-to-date

When we look at a table of percentage differences like this, the idea is to draw conclusions about which performance measures need attention based on the size of the percentage differences highlighted. In this example, two typical comparisons are being made: how this month compares to last month, and how this month compares to the monthly average for the year to date. But do we rely on the bad result of this month being 14.5% worse than last month? Or do we rely on the good result that this month is 48.1% better than the average for the year to date (YTD)?

That’s the problem with monthly comparisons—and with comparing this year to last year, this quarter to last quarter, this week to last week, even this week to the same week last year or this quarter to the same quarter last year. All of these are ‘point-to-point’ comparisons. We are simply comparing two values to each other—a sample of two representing a time series of possibly dozens. And when we do this, we will almost always get conflicting conclusions. Why? Because of routine variation: the fact that things always have some amount of up and down variability due to inherent complexity rather than any kind of real change. The difference between two months or two weeks or two years is a result of both routine variation and real change (if real change is present). Unless we know the size of the routine variation for our measure, we can’t detect the size of any real change if indeed a real change is present.

FIGURE 30. Stacked line chart
FIGURE 30. Stacked line chart for Fluoride Hours Above Maximum

COMPARING YEAR ON YEAR

Another common analysis method is to compare the patterns in a performance measure by calendar or fiscal year. These comparisons are usually presented visually in the form of stacked line or bar charts.

Looking at the graph in Figure 30, how long does it take to get a feel for how performance is currently going for Fluoride Hours Above Maximum?

And what conclusions did you draw? Can you explain what’s going on? What do you predict for the coming months’ values for Fluoride Hours Above Maximum? You probably determined that the past few months of the current calendar year are generally better than previous years, but overall this year isn’t quite as good as last year.

How does this story about the performance of Fluoride Hours Above Maximum compare to the story you drew from the point-to-point analysis in Figure 29?

TREND OVER TIME

To overcome the confusion and overwhelm of trying to interpret signals from noisy up-and-down measure values, tidy-looking analysis methods such as trend lines are used. Often they span a year or two’s worth of measure values, as is the case for Fluoride Hours Above Maximum in Figure 31.

FIGURE 31. Trend line
FIGURE 31. Trend line for Fluoride Hours Above Maximum

How would you explain what this trend line is saying about the performance of Fluoride Hours Above Maximum over the past few years? If you took it at face value, the trend line says there’s been a steady increase. And what does it predict for the future at face value? That we’ll continue to see increasing values of Fluoride Hours Above Maximum? But if we computed a trend line just for the current year (a common strategy) we’d draw a different conclusion. Take a look at Figure 32.

With this trend line, most decision makers would not question the suggestion that Fluoride Hours Above Maximum are decreasing. However, the R² value for this trend line is 0.0492, which means only 4.92% of the variation in Fluoride Hours Above Maximum is explained by the trend line.

Let’s take a closer look at how R² values work. An R² value is a statistical measure of how much variation the trend line explains in a set of data. If we had a performance measure that had a value of one in the first month of the time series, a value of two in the second month, a value of three in the third month, etc. we would have a perfectly straight line in our graph, and a trend line through that perfectly straight line would have an R² value of 1. This means the trend line explains 100% of the variation in the data because the passage of time is perfectly correlated to changes in the data. But almost all data have variability, so a trend line—or the passage of time—will explain only some of that variability. Most trend lines I’ve seen on performance measure graphs explain less than 10% of the variability. Could we really draw a safe conclusion when our analysis is explaining so little of the data’s variability? For the current year trend line for Fluoride Hours Above Maximum we could, at best, say performance is declining only slowly.

FIGURE 32. Trend line for current year
FIGURE 32. Trend line for current year of Fluoride Hours Above Maximum

And aside from this ill-informing trend line, how does this conclusion compare to the conclusions you drew from the first two analyses?

MOVING, ROLLING OR RUNNING AVERAGE

To overcome the assumption of straightness in linear trend lines, as well as smooth out seasonal or cyclical variation in performance measures, moving or rolling averages are sometimes employed. Often they are 12-month moving averages. For example, the value for April 2010 is computed as the average of the individual monthly values from May 2009 through April 2010. The value for May 2010 is computed as the average of the individual monthly values from June 2009 through to May 2010, and so on. This averaging has a smoothing effect, as we can see in Figure 33 for Fluoride Hours Above Maximum.

This chart of Fluoride Hours Above Maximum shows a smooth gradual decline, followed by a subtle increase over the past year or so. Maybe there is even suggestion of a recent improvement since July 2012? No messy month-to-month variation to distract and confuse us. However, is that really the right conclusion to draw about Fluoride Hours Above Maximum? For starters, recall that moving and rolling averages are used to smooth seasonal variation. But does the measure Fluoride Hours Above Maximum even have a strong pattern of seasonal or cyclical variation? Given the number of moving average applications on measures with no seasonal pattern I’ve come across, this doesn’t seem to be a question many think to ask.

FIGURE 33. Moving average
FIGURE 33. Moving average for Fluoride Hours Above Maximum

In each of the above four analyses, what conclusions did you draw about Fluoride Hours Above Maximum?

You almost certainly ended up with contradictory conclusions, ranging from performance getting worse to performance getting better. How alarming is that? The conclusion you ended up with was entirely dependent on which analysis method you were given. That’s no way to manage performance. It’s not acceptable that, depending on how we analyse a performance measure, we can arrive at diametrically opposed conclusions. Does this example make you curious about how many times your decisions, actions or investments actually made the performance of your organisation worse without you even realising it? Does it make you feel a little uncomfortable that analysis methods so widely used and unquestioningly accepted can dupe you so easily? It’s little wonder that the phrase ‘lies, damn lies and statistics’ exists.

The fact that each of these traditional performance management analyses gives us completely different signals about what performance is doing is not because statistics produce lies. It’s because these analyses are based on assumptions that have little statistical or logical integrity in the context of monitoring performance.

Assumption 1: In ‘same month’ comparisons, last year was normal

When we take a look at performance this month compared to performance in the same month last year and say ‘Well, things seem to have gotten worse (or better)’, we’re assuming we are comparing current performance to a reasonable baseline. But we’re not. Last month, or the same month last year, or the monthly average year-to-date are not reasonable baselines.

How do we know last November was normal and a sensible baseline against which to draw conclusions about this November? It makes little sense to think every November should perform the same. To expect November’s variation should be within 5% (or an equally arbitrary 3% or 10%) isn’t based on any sound reasoning.

The same goes for comparing this month to last month. What makes last month an appropriate baseline to evaluate this month’s performance? What if last month was unusually good, or abnormally bad? Do we really think every variation is a signal of change? Everything varies routinely to some extent, even in a stable and unchanging system. The real question should be, ‘How can we know when the pattern of routine variation has changed?’

Assumption 2: If Excel can calculate a trend line for a set of data, then there must be a trend

Trend lines are calculated mathematically in a way that places a straight line as close as possible to the middle of a smattering of points in a time series. How far away the points are from the trend line is an indication of how reliable the trend line is in explaining the variation of the points over time. A large variation means little reliability. And little reliability means no real trend, even if a line for one has been drawn across the chart.

Using trend lines without using the R² statistic mentioned earlier is like driving a car with no appreciation of road rules. The R² statistic is a measure of how well the trend line explains the patterns in a set of data. As mentioned earlier, the R² value can range between 0% and 100%, and we can interpret it as the proportion of the variation in the data explained by the trend line. If we have an R² that is low, then the trend line is explaining very little. If we have a high R² value, then the trend line is a good summary of the predominant pattern in the data. In Microsoft Excel and any other application that computes trend lines, we can add the R² value to the chart as an option.

I think it’s malpractice to present a trend line without the R² statistic. Most trend lines explain very little in our performance measures, and mislead decision makers with overly simplistic, weak, and often incorrect explanations of what performance is doing. Just because our software can draw a trend line through a time series of performance values does not mean a trend actually exists. A trend line is just a mathematical calculation. We calculate a trend line for almost any group of data points.

There’s another reason why trend lines are next to useless for interpreting performance measures. Why would we assume all change happens gradually, and at the same consistent rate? How can a trend line tell us if our performance measure experienced a sudden upward or downward shift in response to an improvement initiative? It can’t. And it distracts our attention away from noticing such potentially important—and very common—signals.

Assumption 3: The world starts anew on the 1st of July every year

Many analyses of performance focus only on the data from the current financial year because we often set targets for the overall end result of the financial year. We rationalise that our attention is on the current year’s plan, and we use the current year’s data to track its progress.

When one month isn’t hitting the target, is worse than last month, or falls behind the expected linear improvement from month to month, we assume there’s a problem. ‘People aren’t working hard enough’, ‘People need education’, ‘We don’t have enough people’ or ‘We don’t have enough money’ are the causes considered most often. And we must throw more effort/education/people/money at the situation next month to make up ground. This kind of band-aid approach to improving performance is endemic: lots of action and effort, but no fundamental and lasting change in performance.

That’s because this year’s performance is not independent of last year’s performance, or the year before that, or the year before that. Business processes don’t recognise the cusp of a new financial year. They just keep on working pretty much the same way they did in the days, weeks, months and even years before. They keep on working the way we designed them to work with our policies, workflows, procedures, and so on. Throwing more effort at a business process that has inbuilt redundancy, waste and rework; is constrained by policies that delay actions; or is ambiguous or ad hoc will never make it perform better for long. We might see a spike of improvement next month, but we won’t be able to sustain the special effort for long, and the performance of that process will soon return to its baseline state. This special effort is only compensating for the poor performance of the process, not removing the causes of poor performance.

We need to use the context of the past to help us see the true capability of our business processes and their current ‘baseline’ performance so we can validly interpret what’s really going on with performance this year and into the future. And we must use this information continuously to check for signals and root-cause-oriented improvement opportunities, regardless of how neatly (or not) they align with our planning timeframes.

Assumption 4: The only probable cause for a difference is that something changed

If we focus on point-to-point comparisons, and our eyes pick up a difference, that difference is not necessarily real. It doesn’t mean that something has changed. In fact, most of the month-to-month or point-to-point variation in our performance measures is simply because we don’t have complete control over every little facet of our business processes and the environment they operate in. And we’ll never have that complete control. Compare the coffee-making process at my favourite local café (The Flying Nun in Samford) with my marketing process to grow subscribers to my Measure Up newsletter.

The variables that affect the making of my most recent half-strength-almondmilk-latte-in-a-mug include only a few things: order accuracy, coffee bean quality, water quality, barista skill, barista attention and coffee machine reliability. These factors are generally easy to control, so my half-strength-almondmilk-lattes-in-a-mug are almost always consistently wonderful. By contrast, the variables that affect the growth of Measure Up subscriptions are many: how diligent I am at finding new networks of potential subscribers, how relevant my content is to readers’ interests, how engaging my writing style is, what else is out there competing with me for the time and attention of potential subscribers, how well my invitation encourages potential subscribers to sign up, and how often my subscription confirmation email will get caught in a spam filter. And that’s just scratching the surface. These factors are much harder to control than the coffee-making process. And so the result I see in the number of new subscribers to Measure Up has greater variability than does the taste of my half-strength-almondmilk-latte-in-a-mug.

The variation in our performance measures is due to many things. Most point-to-point differences are not the result of a change. It’s just routine variation in stable systems and processes. We need to learn how to filter the routine variation, so we can see the signals of real change.

Assumption 5: All changes happen smoothly—at least for the data we choose to put moving averages through

The purpose of moving or rolling averages is to smooth out variation (particularly seasonal or cyclical variation) so we can easily see long-term trends. Rolling averages are capable only of picking up gradual trends, not sudden shifts in the level of performance or several different types of signals occurring close together. Rolling averages are not capable of identifying when a shift or trend began or finished. But we need to know these things—the type of signal, the size of the signal, the start and finish of the signal—because they are clues to the cause of a performance change, and the actions we ought to take in response.

Moving or rolling averages mask true signals in our performance measures. In fact, any analysis that smooths out or removes variation from our data poses a danger to valid interpretation. We need to see our performance measure’s routine variation to assess whether there is a real change and, if so, what kind of change it is.

We can see signals only when we have filtered out the noise

The common thread among these five dangerous assumptions is that none of them help us sort out the two types of variation in our performance values. Variation, first of all, is simply the up and down movement of our performance values from month to month (or week to week, or period to period, depending on how often we measure them). The first type of variation our measures will exhibit is routine variation. Routine variation always occurs, even when there is no real change in performance. It’s just random noise resulting from the inherent complexity of our business processes.

The second type of variation our measures can exhibit is abnormal variation. Abnormal variation occurs only when a change has happened outside the usual operation and environment of our business processes, making performance better or worse. Abnormal variation is what true signals of change are made of.

Most people don’t seem to understand this concept of variation. They treat each performance value as though it were pure truth about the current level of performance. And they treat a difference between successive performance measure values as though it were pure truth about the change in performance. But they’re wrong. They might have learned some statistical techniques at university, business school or on the job. But just because they know what a trend line is, or can calculate a percentage or average or forecast, doesn’t mean they know what they’re doing with their performance measures. It doesn’t mean they’re using the best approach to understand change over time in their measures.

MOST PEOPLE USE STATISTICAL TECHNIQUES WITHOUT USING STATISTICAL THINKING

The science of statistics is fundamentally about separating signals from random, routine noise. It is the study of the variation in the world—the messy, unruly, defiant, complex and often-hard-to-predict results. Statistics is about finding patterns in chaos that help us understand more, and take more informed decisions and actions. So long as we use data that are sufficiently reliable and representative of what we’re measuring, statistics can give us actionable knowledge despite the complexity and chaos of the reality those statistics reflect. But we also need to appreciate that statistics does not give us precise or exact answers. It’s not like pure mathematics or physics. Statistics gives us estimates, generalisations and probabilities that must be interpreted in the context of variation. That realisation is an essential part of statistical thinking. Texas statistician Samuel Wilks (1906–1964)⁴¹ said this in a 1951 presidential address to the American Statistical Association:

‘Statistical thinking will one day be as necessary for efficient citizenship as the ability to read and write.’

Our organisations and businesses are part of the real world. We can’t really manage their performance using performance measures unless we use statistical reasoning with statistical techniques. The common interpretation methods discussed above don’t use valid statistical thinking or reasoning with the statistical techniques they are based on. We need a method that does. And, happily, it’s not that difficult.

Here’s how Step 7 in the PuMP Blueprint does it.

Step 7.1: Measure frequently enough

Step 7.2: Gather enough context

Step 7.3: Use a time-series graph

Step 7.4: Filter out noise

Step 7.5: Look for signals

The tool we are going to use as the foundation for Step 7 in the PuMP Blueprint is the XmR chart. XmR charts are a type of statistical process control (or process behaviour) chart, invented in 1924 by a statistician named Walter A. Shewhart. But while the tool isn’t new its application for performance management as PuMP Blueprint Step 7 describes certainly is.

XmR charts monitor the outputs of processes to provide a rapid and reliable means of detecting change. When we want a process to perform consistently, we want to be aware of changes we didn’t plan for. And when we want a process to perform better, we want to be aware of the impact of our actions on the outputs of the process.

Shewhart invented XmR charts to monitor the production quality of underground transmission equipment at the Western Electric Company.⁴² In doing so, he laid the foundation for quality management based on statistical process control charts for manufacturing and industrial processes in companies and organisations around the world. These charts silently rocked the world. They’ve been used in medical research for monitoring blood pressure of patients; for monitoring camshaft bearing diameters in car engines; and for controlling food safety on the production line. The examples are endless.

What many managers today still don’t realise is that Shewhart’s approach to analysing data for manufacturing performance is equally applicable—and equally transformational—for non-manufacturing performance. We can use Shewhart’s charts to monitor profit, revenue, expenses, staff turnover, on-time delivery, budget variation, customer satisfaction, employee engagement, sales, and lost time injuries. In fact, we can use Shewhart’s charts for every kind of performance measure. Dr. Donald Wheeler, a renowned statistician of our time, wrote Understanding Variation: the key to managing chaos on this very topic. Wheeler introduced the use of the XmR chart for management data (or as we call it, performance measurement). His closing remarks in this best-selling management book are so very true:

‘Millions of people have proven, by their own experiences, over the past 60 years, that Shewhart’s control charts work. This approach to understanding and using data is not on trial. The question is not whether or not the techniques will work—but rather whether or not you will make them work.’

Through Step 7 of the PuMP Blueprint, we can make XmR charts work to transform the value we get from our performance measures by taking a deliberate and sensible (and easy) approach to interpreting their signals.

The 5 steps for interpreting signals from performance measures

Step 7.1: Measure frequently enough

Annual measures are common, often because we try to align measures to our annual strategic and operational plans. If we are going to test whether we’ve achieved our goal of increased customer satisfaction in 12 months’ time, then we plan to measure customer satisfaction in 12 months’ time. But there’s a big flaw in this thinking.

Measures that monitor after the fact give us too little information too late. We get no feedback while we are implementing our plans to increase customer satisfaction. And without that information, we get no clues as to whether those plans are working or not. We miss out on the opportunity to fine-tune our plans to ensure they actually do produce the increased customer satisfaction they were supposed to. It’s a trap to assume our strategies will unquestionably work.

Good performance measures are course-correcting feedback, not after-the-fact proof. We need to track them often enough for the feedback to guide our decisions and actions to improve performance. The definition of ‘often enough’ depends on the timeframe we can expect to get the result we are planning for. If our goal is to change our organisation’s culture, then we are looking at a slow change. Measuring employee engagement daily or weekly would be a waste of time and effort. But measuring quarterly could be useful if our plans might have a detectable effect that frequently. If our goal is to increase new website visitors, that’s something we can affect quickly. Measuring new website visitors daily or weekly probably isn’t too frequent, and would give us the feedback we need to monitor and evaluate the impact of different marketing strategies we could implement week by week.

The idea of measuring frequently enough is that we strike a balance between getting enough data so our measure is sensitive enough to pick up signals, but without overinvesting time and effort in getting more data than we need. Until we find that balance, it’s better to measure a little too frequently than not frequently enough.

In our earlier example, TVWD’s Fluoride Hours Above Maximum was measured monthly. That’s a common frequency for most performance measures because performance reports are usually produced monthly.

So the first step toward making our performance measure easier to interpret validly is to measure frequently enough that we don’t dilute the signals.

Step 7.2: Gather enough context

Comparing this month to last month is not enough, mostly because it’s a sample of size two from which we are trying to draw conclusions about variability or change. That’s a small sample.

Without a doubt, the best kind of performance information is trend over time. A trend over time puts the data of a performance measure into an appropriate context from which we can draw statistically valid (or in everyday language, credible) conclusions. If we just look at this month’s performance we’ve got nothing to compare it to, and so we’ve got absolutely no way of knowing whether it’s good or bad. If we just look at this month’s performance compared to last month, or compared to a target, or compared to the same month last year, we still don’t know whether the difference we see is within the routine variation typical of our measure or a real signal of a change in performance.

The operational team at TVWD had 20 years’ worth of measure values for Fluoride Hours Above Maximum. That much history is not usually necessary unless we have an insatiable curiosity to see how our measure has been tracking since its birth. Generally anything older than four or five years is not really going to inform actions we might take in response to current performance. Two or three years is just fine in most cases. And that’s what TVWD chose for Fluoride Hours Above Maximum: the past three years’ monthly values, which we can see in Table 21.

Although two or three years of a measure’s values is practical, we still need a bare minimum of five consecutive values of a measure just to get started establishing a baseline to represent the current performance level. Sometimes we might need even more than five values to establish that baseline. Only when we have a baseline do we have a chance of making sensible interpretations about whether performance has changed or not. And once we have our baseline, we need at least another three measure values, and often more than eight, before we can be sure whether performance has changed or not.

Step 7.3: Use a time-series graph

Tables will not show us patterns in our performance measure data. Remember the experiment we did at the beginning of the chapter? At the very least, we need to make sure our performance measure values are plotted in a line chart to make visual detection of the patterns over time quick and easy.

We can easily see a pattern when we look at Fluoride Hours Above Maximum in a time-series arrangement such as the line chart in Figure 34. Some might argue that a bar chart would be just as useful as a line chart. If so, I think there’s a chance they are just defending what they’re already doing. Remember Edward Tufte’s data-ink ratio? The line chart uses far less ink than a bar chart, and conveys the message much more clearly.

Step 7.4: Filter out noise

Sometimes signals are very prominent and easy to pick up with our eye, like the shift we can see in Fluoride Hours Above Maximum in Figure 34. But not all signals are so convincingly obvious. Sometimes signals are more subtle, and our eye doesn’t easily distinguish them from the routine variation in our measure. Other times our eye will trick us into thinking there’s a signal when in fact there isn’t.

We might, for example, look at the tail end of the line chart for Fluoride Hours Above Maximum and see the start of an increasing trend. Don’t be fooled. We still don’t know enough about the routine variation in Fluoride Hours Above Maximum, so we can’t reliably decide whether this little upward kick is a part of that routine variation or not. What we need is a way to gauge—or measure—the size of the routine variation. And that’s where XmR charts come in.

XMR CHARTS: THE BEST WAY TO DISPLAY PERFORMANCE MEASURES

An XmR chart is a very specific type of statistical process control chart that is fundamental to finding the true signals of change in performance in our measures. Don’t groan. You’ll see very quickly just how easy and powerful it is. When I teach this method in my PuMP Blueprint Workshops, even people who consider themselves statistically illiterate and fearful of maths love it. I regularly witness the most numerically averse people correctly create their first XmR chart, validly interpret it, and feel very excited about using XmR charts from then on. It will be well worth our time to learn how to create them. For now, let’s take an introductory look at what the XmR chart is, and how to use it to filter out the noise from our performance measures so we can more easily see their signals. But I also recommend reading Donald Wheeler’s book, Understanding Variation, to deepen your knowledge. Or you can take my online course ‘How to Get the Truth Out of Your KPIs Using Smart Charts’.⁴³

FIGURE 34. Line chart
FIGURE 34. Line chart of Fluoride Hours Above Maximum

The ‘X’ in XmR is the symbol that represents our performance measure, and the ‘mR’ represents the moving range calculated from the differences of successive values in the performance measure time series. The ‘mR’ is how the amount of routine variation in our performance measure is calculated. So the XmR chart is actually two charts, as we can see in Figure 35.

The first of the two XmR charts displays the values for TVWD’s performance measure Fluoride Hours Above Maximum. There are three essential parts to the anatomy of this first graph, known as the X chart. The first part is the line that plots the performance measure values. We can use XmR charts with as few as five or six performance measure values in a time series. But the more historical values we have, the more context we have for interpreting what performance is doing now. If we’ve got only three or four measure values, we really can’t use one of these charts because we haven’t got enough data yet to reliably detect real signals. So we need to wait until we have a few more performance measure values. The values plotted in this first graph in the XmR chart must be actual values for each time period. The actual Fluoride Hours Above Maximum for each month—not a moving average or year-to-date figure—is plotted in Figure 35.

FIGURE 35. An XmR chart
FIGURE 35. An XmR chart for Fluoride Hours Above Maximum

The second part of the X chart in an XmR chart is the central line. It’s the flat line going through the middle of the Fluoride Hours Above Maximum values in Figure 35. In general, the central line is calculated using the first five or so points that we plot in the chart for the first time, and we don’t recalculate that central line unless a signal suggests that we need to. (We’ll come to this in Step 7.5 of the PuMP framework for Interpreting Signals from Measures.) If the central line is calculated to be 40 from our first five or so measure values, then that central line stays at 40, even as we add more performance measure values to the chart. The central line is our baseline, and it needs to stay the same so we have a consistent reference point for deciding whether and when performance is behaving differently. We recalculate the central line only when our performance measure has shifted to a different level and we need a new baseline to look for future changes in performance. In Figure 36 we can see that the next several months’ of values for Fluoride Hours Above Maximum have been added to the XmR chart, but the central line still remains at 40. These new points do not signal change, as we’ll learn when we reach PuMP Step 7.5. The baseline is still a good description of the current average level of performance.

The third part of the X chart in an XmR chart is a set of two lines called the upper and lower natural process limits. Natural process limits are the gauge of a performance measure’s routine variation. They are not a guideline for where our performance measure should be, or targets, or standards, or limits of acceptable variation. They are calculated from our performance measure’s data. Targets and standards are different from natural process limits, but as we’ll see in PuMP Blueprint Step 8 targets can work brilliantly with XmR charts.

The mR chart portion of the XmR chart plots the measure’s ranges. It’s based on successive differences between the measure’s values. For our Fluoride Hours Above Maximum example, the first point in the mR chart is calculated by taking the absolute difference between the first two values of Fluoride Hours Above Maximum. The first two values (from Table 21) are 54.63 and 33.60 and the difference is 21.03, which becomes the first point plotted on the mR chart. The next value that would be plotted on the mR chart would be the difference between second and the third value in the Fluoride Hours Above Maximum time series, which produces 8.56. Just as the X chart of the performance measure’s values has a central line and natural process limits, the mR chart has an average moving range line and an upper moving range limit. Because the moving range values can never be negative and have a natural lower limit of zero, we won’t see a lower moving range limit.

FIGURE 36. Adding new measure values
FIGURE 36. Adding new measure values to the X chart of Fluoride Hours Above Maximum

The mR chart is great at telling us when the amount or pattern of variability in a performance measure’s values has changed. Variability is important not only for discerning signals from noise, but also for highlighting how much control we have over a performance measure. When a measure’s routine variation reduces, performance is more predictable. Usually, a lot of routine variation is bad for performance. For Fluoride Hours Above Maximum, the mR chart (in Figure 37) shows that the amount of month-to-month variation in the measure is fairly stable, with no real changes. There is one value that’s unusually high (it falls above the upper moving range limit), but coupled with what we see in the X chart there isn’t strong enough evidence that it’s a signal.

A FINAL WARNING

Don’t make up your own calculations for the central line and natural process limits in XmR charts. There is a correct way to calculate them from the data for a performance measure, developed by statisticians who understand the statistical concepts that underpin XmR charts. We need to learn the correct method and stick to it. Again, I refer you to Donald Wheeler’s book, Understanding Variation, for the best layman’s explanation of how to construct XmR charts correctly, or my online course ‘How to Get the Truth Out of Your KPIs Using Smart Charts’. And then go to this book’s website (www.pumpblueprintbook.com) and download an Excel spreadsheet template to help you quickly and easily create XmR charts for your performance measures.

FIGURE 37. The updated mR chart
FIGURE 37. The updated mR chart for Fluoride Hours Above Maximum

We can use XmR charts for virtually any kind of performance measure.⁴⁴ Using both charts that make up an XmR chart provides the right kind of information to help us see real signals of change in our performance measures. That means we can respond when we need to, and not respond when it would be a waste of time and effort. The central line and natural process limits are the keys to highlighting the real signals and stopping us from making two mistakes:

Interpreting noise as a signal (thinking a change happened that actually didn’t)

Interpreting a signal as noise (not noticing a change that actually happened)

Avoiding these two mistakes means knowing how to recognise true signals in our XmR charts.

Step 7.5: Look for signals

The signals we look for in our performance measures are not just about steady linear change over time. There are also sharp upward or downward shifts in performance, as though something has changed overnight and performance moves to a completely different level. Sometimes the shifts are very big and noticeable, but sometimes they can be very small and subtle. And unless we’re using an XmR chart, we’d miss the signal completely. There are also signals of outliers: single performance measure values that fall outside the range of routine variation (i.e. outside the natural process limits). The way we should respond to these outliers is often completely different from how we respond to other signals. Defining real signals to look for in our performance measures is what this final step of the PuMP Measure Interpretation technique focuses on.

Look what happens in Figure 38 when we add the next several measure values to the XmR chart of Fluoride Hours Above Maximum.

These four most recent points in the X chart of Fluoride Hours Above Maximum are all dramatically below the central line. That’s a signal that performance has changed, and that Fluoride Hours Above Maximum has decreased. It’s called a short run, and it’s indicative of a change that had a fairly sudden effect rather than a gradual effect. Something happened around October or November 2010 that fundamentally changed the number of hours per month that fluoride levels went above the maximum ideal level. Whatever the change was, it means the performance of Fluoride Hours Above Maximum is now different. When we get certain types of signals, such as a short run, we often need to change the central line and natural process limits to reflect the new level of performance. In other words, we need to set a new baseline to describe the level of performance we now have.

In the case of Fluoride Hours Above Maximum, the signal comprised only four points. Before recalculating a new central line and natural process limits I prefer to wait for another couple of performance measure values, and then use the five or six values from the start of the signal for the calculations. We can see how this looks in Figure 39.

That’s a big change, isn’t it? The central line shifted down from 40 to only 3.7. What could cause such a rapid and large change in performance? In this case, the change was due to the U.S. Environmental Protection Agency lowering its recommended level of fluoride in drinking water from a range of 0.7 to 1.2 milligrams per litre (mg/L) to a maximum of 0.7 mg/L. That change took effect in January of 2011, so the target dropped at that time and immediately stabilised with just a few hours per month over that level. Obviously the water district’s ideal is to have zero hours above maximum. But that’s the nature of performance measures: we are measuring a result we can’t yet sustain at the level where we want it to be. Unfortunately, lowering the recommended level of fluoride in drinking water was not enough to improve the system permanently. As we can see in Figure 40, in the months following the recommendation and change, Fluoride Hours Above Maximum gradually increased again, both in average level and variability. It is difficult to set a new baseline for Fluoride Hours Above Maximum over this time because of the chaotic fluctations. The system for managing fluoride levels would need to be stabilised to remove this chaotic variation.

FIGURE 38. Adding the next measure values
FIGURE 38. Adding the next measure values to the XmR chart of Fluoride Hours Above Maximum

FIGURE 39. Setting a new baseline
FIGURE 39. Setting a new baseline for Fluoride Hours Above Maximum

FIGURE 40. Fluoride Hours Above Maximum
FIGURE 40. Fluoride Hours Above Maximum after the recommended fluoride level changes

The short run signal we looked at originally for Fluoride Hours Above Maximum is only one of the possible signals an XmR chart can show us. Let’s briefly look at the other signals.

THE SIGNALS TO LOOK FOR THAT TELL US PERFORMANCE HAS CHANGED

XmR charts come with very clear guidelines about the types of patterns that indicate a performance measure is no longer following that random pattern of routine variation, and that its baseline (the combination of the central line and the natural process limits) is no longer an appropriate description of the current performance level. The basic types of signals are summarised in Table 22.

The first signal type is an outlier, often referred to as a special cause. This is when one point in our performance measure’s time series behaves dramatically differently from the rest. In an XmR chart, it’s when a point falls outside (either above or below) the natural process limits. Usually a special one-off event has happened to cause that aberration. We don’t need to fix anything—the event is not likely to happen again anytime soon, and so devising a solution would be a waste of time and effort. But it’s good to know what happened. If we were to see a value of Fluoride Hours Above Maximum fall below the lower natural process limit, it might be caused by something like a new engineer misreading the fluoride instructions and adding much less fluoride than required. This is not likely to keep happening because someone would pick that up pretty quickly through routine monitoring and testing, and the new engineer would quickly correct the mistake.

The second signal is called a long run, in which seven or more consecutive points sit on the same side of the central line, either all above or all below. The number seven is the minimum number we need to be convinced this is a real signal. I realise that sounds like a lot, but it’s important to remember we want to avoid over-reacting and fixing problems that aren’t really there. So we look for seven points in a row, or 12 out of 14 points in a row (two of the points in the row of 14 could in this case be on the other side of the central line from the remaining 12).

The third signal is the short run, as we saw in the Fluoride Hours Above Maximum example. To qualify as a signal, a short run requires only three out of four points in a row to be on the same side of the central line and closer to a natural process limit than they are to the central line. Usually short runs are due to improvements or problems that have a substantially large impact. Of course we’ll want to find out why this happened and then decide what the best thing is to do about it.

TABLE 22. The three basic signals to look for in an XmR chart
TABLE 22. The three basic signals

However, before we can reliably detect signals, we first need to have some consistent level of control over the performance result we are measuring. Absence of control produces a chaotic pattern of variation. When variation is large and all over the place, it’s a sure sign we have little or no control over the performance result we are measuring. Often that’s because we don’t have any systematic method of producing that result. A chaotic pattern in Fluoride Hours Above Maximum would be a sign there was no systematic approach to testing and adjusting fluoride levels in the water to a sufficient degree of accuracy. The first thing to do when we have signs of chaotic performance is to look at the business process, and decide how to streamline and standardise it so we have less variability and greater predictability in our performance result. Then we can see real signals.

WHY DO WE NEED AT LEAST SEVEN POINTS TO QUALIFY AS A SIGNAL?

Sometimes we might hear that six points in a row is enough to qualify as a signal, and sometimes eight points.⁴⁵ I don’t know why the rules vary like this. The reason we need more than two or three (or even five or six) values before we declare a signal has to do with the probability that the pattern we see is completely random—part of routine variation—and not caused by any particular change. First, we need to accept this idea of routine variation and randomness. No performance measure has zero variation. If a measure did have zero variation it would be perfectly predictable, and we would have complete control over those performance results. And if we have complete control and perfect prediction, it’s not a result we should bother monitoring with a performance measure. So let’s operate from the premise that all performance measures will have some element of natural and routine variation over time from week to week, month to month whatever frequency we measure. Now we need to talk about probabilities.

Imagine we have a measure of Revenue we’re monitoring monthly. We’ve done nothing to try to improve revenue for a year or two, and nothing major has happened in our business environment to change it either. So Revenue will exhibit only routine variation overtime, bouncing up and down randomly from month to month. When the variation is random like this, there is roughly a 50% chance that April 2013’s revenue will be larger than March 2013, and roughly a 50% chance that it will be smaller. No-one in their right mind would assume this was evidence of a problem or an improvement. If we did, we’d be reacting every month and making things worse by adding another source of variation: layering on our own ad hoc ‘fixes’ that would just increase the complexity of our revenue-generating process (known as ‘tampering’). The probability of April 2013 and May 2013 both being larger than March 2013 is 25%.⁴⁶ That’s still a large enough probability to call the pattern a part of routine variation. We can’t confidently declare that revenue has improved. And when we see that June 2013’s revenue has dropped back down again, we realise revenue most definitely hasn’t improved. Even three months in a row larger than March 2013 still has a fairly large probability of being part of routine variation: 12.5%. A signal is something that falls outside the pattern of routine variation, and it needs to be convincing. Seven points in a row forming a long run has a probability of 50% × 50% × 50% × 50% × 50% × 50% × 50%, which equals 0.8%. Patterns like these have less than a 1% chance of being completely random, or of being part of routine variation. Or to put it another way, there is a 99% chance they’re not part of routine variation and have an assignable cause. That makes them a signal.

XmR charts do not only have a big bearing on how we interpret our performance measures so that we respond only to signals and not random noise. They also have implications for how we set targets. Are you curious about how exactly to specify a target when you’re using XmR charts, and why those targets are so much more powerful than the annual targets you’re setting now? Hang around for Chapter 11.

Patterns, not points; signals, not noise

Our new mantra when looking at any performance measure’s values to determine whether performance has changed at all should be ‘Patterns, not points; signals, not noise.’

The idea of Step 7 of the PuMP Blueprint is to reinvent the way we look at our performance measures and hear what they are really telling us. By learning how to think statistically, and not just using statistical techniques, we’ll get so much more insight from our measures. Specifically, we’ll stop wasting time and money on fixing things that aren’t broken, and use less time and money fixing things before they break completely.

Take Action: Create XmR charts and interpret your measures

To get started quickly, and develop a catalyst for evidence-based management, set up your performance measures in XmR charts. Go to the book’s website (www.pumpblueprintbook.com) and download the PuMP XmR Excel spreadsheet template and accompanying XmR Chart Instructions in Adobe PDF format. If you want more confidence and skill with using XmR charts for performance measurement, visit www.usingsmartcharts.com.

Chapter summary

We need to stop the bad habits of comparing this month to last month, trusting linear trend lines, and using moving averages for our performance measures.

We need to base interpretation of performance measure signals on statistical thinking, not statistical techniques alone.

Using XmR charts for our performance measures will show us what is really going on with performance. Even if we hate statistics, or believe we’re statistically illiterate, we need do this. The benefits of valid interpretation of performance far outweigh any pain we must work through during the 15 minutes it takes to create an XmR chart and start using it.

When your Measures Team has at least one of their new measures set up in an XmR chart, it’s time to ramp up the momentum and start using that measure for real performance improvement.

11

Step 8: Reaching Performance Targets

‘[P]eople spend too much time aiming at the bulleye and not enough time shooting at it. Rather than placing so much emphasis on getting ready and aiming, go ahead and take a shot. Taking the shot gets you started and also lets you gauge how far off the mark you are. Make adjustments, but keep shooting until you get closer and closer, and eventually you will hit the bulleye.’

— Jason Selk, 10-Minute Toughness

‘Measurement without the opportunity to improve is harassment!’

— W. Edwards Deming

We’re at the pointy end of the performance measurement process now. It’s the end many people never reach because of a lack of momentum, discipline or vision. It’s the end where our performance measures produce the goods. It’s when we get a lovely return on the investment we’ve made in those measures by using them to improve performance. That’s the whole purpose of performance measures: to make the right things better.

Most performance measures never improve performance

We don’t get a return on our investment in the performance measurement process until they have taken a lead role in helping us actually improve performance. There’s no value in measuring customer satisfaction if that measure never plays a role in making the service experience for customers better, the products better, or our relationship with our customers better so they are happier.

Even if we now have fabulous performance measures after following the first seven steps in the PuMP Blueprint, we’re still not out of the woods yet because performance measures rarely cause things to get better spontaneously. Sometimes they do. Just the act of measuring an important result, such as employee stress or burnout, can trigger people to focus on the result and look for solutions. But more often than not shifting performance requires a deliberately orchestrated effort.

That orchestrated effort requires removing the following obstacles that are ingrained in most organisations:

Choosing initiatives and targets before choosing measures

Not having a routine forum for using measures to manage performance, so the measures don’t fundamentally inform decisions

Jumping to premature conclusions about what performance measures are telling us, so we respond when we shouldn’t and don’t respond when we should

Mistaking excuses for reasons about why performance isn’t improving despite our efforts so far, and giving up too soon

Using performance measures to judge the performance of people rather than inform people about the performance of the processes they are working in and on

Lacking discipline in improvement and applying easy fixes that only compensate for poor performance rather than fundamentally lifting performance capability.

You might not think these obstacles are yours. But let me offer a perspective on each of them before you decide for sure.

Choosing initiatives and targets before performance measures

A weird thing happens in many organisations’ planning processes. They choose their strategic goals, their strategic initiatives, their targets, and finally (usually) much later their performance measures. Often there won’t even be a performance measure until after the plan is published. A typical plan looks like Table 23.

I’ve seen this time and again. It’s weird because logically we can’t choose a target until we know what we’re measuring. How can we set a target for the efficiency of recruitment until we have decided how to measure that goal? Saying that our target is a 10% improvement is stabbing in the dark. How do we know 10% isn’t too much, or isn’t even enough of an improvement? We don’t know the measure yet, so we don’t know our baseline—the level of performance we’re starting from. Setting a target of 100% is just as ludicrous. One hundred percent of what, exactly? And do we know what that level of perfection is going to demand in effort and investment? This is why so many people freak out when they are given targets for perfection. They start thinking of all the things outside their control—even beyond their influence—that they know are currently keeping performance below 100%.

It’s much more sensible (and inspiring) to set a target after we’ve decided the best measure of the goal to ‘Enhance efficiency of recruitment’ is Average Days to Fill Vacant Positions, established that the baseline is currently 93 days and worked out that to meet other business goals, we really need to reduce that number to 45 days. A target of 45 days is a meaningful, tangible target. It will make sense to everyone, and will be likely to inspire people to think about what they can do to reduce the cycle time instead of thinking about all the things beyond their control and influence.

It also makes little sense to choose an initiative before we understand what it will take to reach the measure’s target (the sensible target, of course). How else can we be confident we’re not over- or under-investing in improvement? How do we know educating recruitment officers is the change that will bring about a big enough improvement in efficiency of recruitment? How can we know the combined impact of the initiatives we need before knowing the size of the gap between actual recruitment efficiency and our desired recruitment efficiency? Apparently, it’s a human tendency to jump to solutions before adequately understanding the problem. So we jump to initiatives before adequately understanding the size and the cause of the gap between as-is performance and to-be performance.

TABLE 23. Typical layout of a strategic plan
TABLE 23. Typical layout of a strategic plan

The more logical and useful order to lay out a strategic (or indeed any) plan is shown in Table 24—results, then measures, then targets, and finally initiatives.

In this layout, the measures are now positioned to play their true role—to act as the feedback that links a chosen action to its intended result. You can find a strategic plan template with this kind of layout at the book’s website (www.pumpblueprintbook.com).

No routine forum for using measures to manage performance

Many organisations still don’t have regular, formal performance review forums as part of their management processes, such as strategy execution and strategy review. And even when an organisation has such a forum, performance measures usually aren’t a formal item on the agenda. How can we think we’re reviewing performance when we’re not using performance measures to give us objective evidence about the results we’re getting from the actions we’re taking? Have you noticed how much of the talk at these meetings is about how initiatives and programs are progressing, and the new challenges and opportunities flying out of left field? Rarely do our performance measures set the agenda for what we talk about in performance review meetings.

If we did have regular, formal review meetings, I think we would take our performance reports and the design of them far more seriously. Not only that, I think we would take the way we interpret our measures more seriously. Often when people are looking at the wrong type of analysis, or they’re all looking at the same analysis but have different ways of interpreting it, most of the time is spent arguing about what the data are really saying, or their integrity. We’re not really using a measure to figure out, ‘What is this telling us about what’s working and what’s not working? Have our priorities changed. Are we on track’?

TABLE 24. Improved layout of a strategic plan
TABLE 24. Improved layout of a strategic plan

A formal performance review forum needs to be regular enough to allow us to respond to the signals in our performance measures before little pains turn into large problems. We need to know quickly if an initiative isn’t working well enough, or if a target has been met ahead of schedule and some re-prioritising is needed. There’s no point having an annual performance review forum just to assess whether targets were achieved or not. By then, it’s too late to do anything about problems. Performance measurement should be a health plan, not a post mortem.

Knee-jerk reacting to data

Managers at a timber company sawmill thought their performance dashboard was the duck’s guts⁴⁷ at first. It tracked a multitude of performance measures to indicate how sawmill operations were going. The data updates for many of these measures was almost live, so their dashboard figures refreshed every few minutes. Traffic lights—red, green and yellow visual flags summarising whether performance is bad, good, or heading toward unacceptable—were also updated each time the data feed refreshed. The managers and supervisors would react to these traffic lights with unnecessarily large interventions, such as changing the settings on timber processing equipment or altering procedures.

These traffic lights were changing based on data that often came from a small sample, such as a single day. Long-term trends and natural variability in the data were ignored. Rather than looking for signals in their data, they were reacting to the noise—to any variation in the data. Everyone was so busy reacting to data that the elements that really needed to be changed were left unattended, and overall performance worsened. Executives eventually realised the dashboard was causing managers and operational staff to continually tamper with the production processes, and removed it from the site. Since then, the sawmill has reduced their workforce from 280 to 180, developed much more clarity about the performance results that matter most, and improved performance in some very important areas. When they were using the dashboard, they had too many people trying to make too many interventions, and a lot of this activity was uncoordinated and unnecessary. The dashboard increased complexity, so variation increased and performance declined.

Step 7 of the PuMP Blueprint, Interpreting Signals from Measures, is essentially about this very problem. Recall that we talked about how performance is always going up or down from day to day, week to week or month to month. No matter how frequently (or infrequently) we’ve chosen to track our performance measures, there will always be variation from point to point. These variations do not mean performance is changing. If we react to one of these random fluctuations, we’ll be wrong at least 50% of the time. That’s because when the fluctuations are random, there’s a 50% chance this month will either be higher or lower than last month. And we’ll be wrong more than 50% of the time because very few fluctuations actually mean a real change happened. Most fluctuations are just natural and random variation. The real signals of change in performance aren’t part of this natural and random variation. They are patterns that stand apart from it.

Do managers really need to look at event-level data? Do they really need to monitor the details of each and every accident, late delivery, lost customer or operational performance measure? These details might be interesting, but how can it be useful for every manager to keep track of every performance-related event? Who really has the time? If one manager is taking care of late deliveries, why would the two managers above them in the hierarchy need to as well? What possible help could they really contribute by duplicating their effort? And if everyone spent time watching over the outcomes of each individual event, who would be watching out for the overall patterns? Those patterns are where we find the insights about how to fundamentally improve performance.

Drawing a conclusion about whether performance has changed by comparing this month to last month is tantamount to making things up as we go along. The insightful conclusions that lead us to act when we should and not act when we shouldn’t come from the patterns in our performance data. They never come from comparisons between the points of our performance data. Comparing points is a deeply rooted habit that must be changed, and we’ll see it do its worst in performance review forums.

Mistaking excuses for reasons

Excuses and reasons are not the same thing, even though they might refer to the same thing. A management team in the vocational education sector I consulted with many years ago had set a goal to grow revenue over a 12-month period. But at no point during those 12 months did we see any kind of improvement at all. They noticed other educational institutes were also experiencing revenue falls, and saw an overall decline in the numbers of people taking vocational courses in their business environment.

This management team had two choices. The first choice was the one they made—use the current business environment as the excuse for not achieving their revenue target and giving up. The second choice would have been to see the immediate reason for their revenue problem was in the business environment and ask ‘What can we do to leverage or respond to what’s going on in the business environment. How can we keep striving for our revenue goal, or perhaps slow its decline if we can’t turn it around’? The first choice is the easy one, but the second choice is the better one.

Excuses are cop-outs that effectively release us from responsibility when performance falls short of our goals. But if we take our goals seriously, surely we’re not going to give up pursuing them at the first sign of trouble? Using excuses to relieve us from responsibility is very often a cultural problem, deep in the psyche of the organisation’s people and the acceptable norms of performance management. I’m not sure whether it’s because of fear, complacency, feeling stuck, all of those things or something else. I do know that people who have an excuse-focused attitude achieve very little, either personally or professionally, while people who have the opposite attitude achieve far more.

The opposite of the excuse-focused attitude is not an accountability-focused attitude. If we dwell too much on the word ‘accountability’ we just deepen the sense of dread that drives people to find excuses in the first place. No, the opposite of this excuse-focused attitude is a solution-focused attitude where we pursue our performance results until we achieve them. And as Jason Selk says in his book 10-Minute Toughness⁴⁸, a relentless solution focus is often needed for high performance. We don’t measure performance for the fun of it. We measure it to dramatically improve it, and achieve lofty goals and targets that mean the world’s a better place from our endeavour. Even though Jason Selk is a sports psychologist, his wise words apply to those of us in business too. We need some toughness to break through mediocrity to high performance. And a relentless solution focus, coupled with a results-oriented focus, is the new cultural norm that we ought to be practicing every day.

Mediocrity is also the by-product of aversion to risk and fear of failure. We don’t excel and achieve great results by working only inside our circle of control or comfort zone. We have to look outside our circle of control, and improve performance by working in our circle of influence. In our circle of influence, other people and forces can impede or facilitate our success in pursuing the results we want. Performance measurement is not about control. It’s about influence, and learning how we can step up to reach results we haven’t been able to reach so far. That’s a much more empowering way to view performance measurement than to think we measure to control. Using the word ‘control’ in this context has given people permission to say, ‘I’m not measuring it because I can’t control it’. With that attitude, they won’t be measuring anything meaningful or worthwhile, and they won’t be achieving anything meaningful or worthwhile either.

Working in our circle of influence will also go a long way towards helping us overcome our blaming and excuse-making habits and relentlessly focus on solutions instead. All forms of blaming must stop: blaming employees, other organisations, the economy, regulations, customers, lack of data or lack of resources. Instead of abandoning responsibility for the performance results we’re getting, we must find the scope of our influence and explore it deeply to figure out what we can do to exercise responsibility.

Judging rather than learning

Quite a few people still seem to think performance measures are needed only at the end of the planning cycle to check whether goals were achieved and strategic initiatives implemented. These people will decide on the performance measures while developing their plan (or soon after), and then after the year has passed will start looking to those measures to see whether the plan was achieved. If we believe performance measures are solely for judging whether or not something happened, it’s understandable that we’d use them this way. But that’s not what performance measures are for. They are feedback to help us execute our strategic initiatives, test whether those initiatives are working or not, change them as the world around us changes (along with all the assumptions we made when we planned), readjust our budgets so that resources go where they are most needed, and learn where the true leverage is for us to achieve the strategically important and mission-critical results. Phil Jones discusses this in his book Communicating Strategy⁴⁹:

‘.  .  .  you may believe you have the best strategy, that you are communicating it in the most congruent and effective way, but it will fail, because the organization’s systems, structures, practices and processes will stop it happening. Failing to address these issues will mean that you effectively handcuff the organization and restrict its ability to execute your strategy.’

Many organisations still use performance measures to judge the performance of people in annual performance appraisal processes. What is the real purpose of employee performance appraisals? Is it to work out which employees need to be retrenched or fired? Or is it to help employees develop and grow their potential as human beings and contributors to the organisation? If it’s the former, then performance measures will always be seen as a threat, and will never get the support required from staff to produce accurate and timely information to managers. If it’s the latter, then what staff do with their performance measures becomes much more important than where those measures position them in the ranking of best performer to worst performer.

I realise performance measures are not the sole source of information for most employee performance appraisal. But their use even as part of the evaluation process puts them in a role far beneath the potential of both the people and the measures. Performance measures are tools for people to use to inform them how they can improve aspects of organisational performance. When measures are instead used as tools to judge the people themselves, the measures create all kinds of unintended but entirely logical behaviours—choosing measures that are easy to game, leaving out data that make the performance measures look anything less than acceptable, and shifting attention to whatever can cover their bums instead of great ideas that can improve performance. The managers at one of the aforementioned timber company’s other main sawmills spent a lot of their time cajoling the rest of the company to change the performance measures used to make that mill’s results look good. For example, they changed a measure called Runtime (the percentage of sawmill operating hours the mill was processing timber) by adding and extending meal breaks and adding breaks for meetings to effectively reduce the total operating hours. This meant Runtime could get closer to 100% (or higher) without any increase in timber production.

Performance measures are feedback about organisational processes and results, not about people. This attitude is important to establish so measures get used for improvement rather than having people fiddle with the figures to make the measures look good even as performance gets worse. I really don’t believe an organisation can successfully use measures to evaluate people’s performance without first having a very strong performance culture where performance measures are used routinely to improve mission-critical and strategically important performance results. This is a culture where people don’t fear measurement, and are good at measuring the right things. They see measurement as a tool to help them do the best job they can, both in the business and on the business. In this kind of culture, people don’t take the measures personally as a threat to their well-being, dignity, job satisfaction, financial situation or career prospects.

Even if a strong performance culture does exist, plenty of research suggests adamantly that what keeps most people happy in their jobs is not performance-based rewards. Things such as liking their jobs, having good relationships with co-workers, having good relationships with their manager, having new and interesting challenges, and feeling valued by the organisation are far more important to most employees than performance-based rewards.⁵⁰ When employees see how their efforts contribute to overall performance of the organisation, they feel a part of something bigger than themselves. This increases their engagement, and improved performance naturally follows.

Treating symptoms rather than causes

A sugar mill was increasing its production, and needed a rail-based freight business to increase its capacity to move more sugar from the mill to the port for export. The freight business looked at its traditional financial reports, carefully ran scenarios with the numbers, and worked out how much they could spend on additional rolling stock and still make a profit. Their solution was to invest millions of dollars on more trains and wagons to meet the demands of increased sugar production. But their solution was wrong. One manager wasn’t convinced that investing in rolling stock was the most meaningful way to do it, and modelled the sugar freight system using Planimate software.⁵¹ The program lets you gather data about how the business process works and then build a simulation of the process. You can then change various parameters to see what effect they might have on a specific performance measure. The model was built with actual data about the length of the track, the schedule of trains, the lengths and capacities of the trains, the variation in the train transit times, the variation in the sugar loading and unloading times, expected delays at the mill and the port, the capacity of ships at the port, and so on. Various solutions were tested using the model, including investing in more rolling stock, to see which solution had the best impact.

The best solution was startling. It required no additional investment in rolling stock, it improved cycle times, met more than the mill’s planned production increases, and even reduced the operating expense for that business process. The solution identified an incredible amount of waste in the design of the freight process, and transformed the process to better serve the customer and produce greater profit for the rail company. The solution was to change the way trains were used to move the sugar from the mill to the port. Rather than doing what railways always did (schedule trains of apropriate length and frequency based on the production numbers the customer provided each week) the solution involved using unit-length trains that cycled continuously from mill to port. And rather than doing what railways never did, these unit-length trains were designed to have one wagon more than they needed as a buffer. This ‘buffer wagon’ became a powerful performance indicator for the system. When the buffer wagon was needed to carry additional sugar, this was a lead indicator that mill production was ramping up again. It gave the railway company time to adjust the system to meet the increased production so freight transport would not become a bottleneck for the sugar mill. The freight process was made fundamentally better because this solution addressed the real root causes constraining performance: outdated policies.

This solution was an innovation because the railway mindset had always been that the only way to increase the capacity of a rail system was to add more trains and more wagons. The Planimate model showed the policies that had always been in place, and the practices that were always accepted as right, were constraining performance. It showed that the capacity of the rail system was constrained by policy, not resources.

Most measures don’t improve performance because we don’t have the discipline to do the fundamental cause analysis needed to drive performance improvement. Our actions focus on easy fixes, such as educating people, writing policies, or buying more trains and wagons. ‘We have to do some customer education.’ We need a policy for that.’ We need to increase our rolling stock by 20%.’ No, we probably don’t. We need to get to the root of what’s going on and focus on the capability of our business processes. How can we fundamentally change the process (the way we do the work and produce the result) to get the outcome we want? If customers aren’t paying our bills, they don’t need education in how to pay bills. We probably need a better billing process that’s easier, quicker, and makes it much simpler for customers to understand how much they’ve got to pay when they’ve got to pay it by and what the payment options are. And it needs to get the bill to them quickly so they’ve got plenty of time to pay.

Process improvement methodologies such as Lean, Six Sigma, and Business Process Reengineering, and modelling tools such as Planimate, couple very well with performance measurement. They provide a disciplined approach to diagnose and improve process capability. They help us to look at how we’ve designed the process for doing the work or delivering the service, and show us how to streamline that process to remove waste and reduce the opportunities for problems to occur. Coupled with our performance measures, process improvement gives us a way to remove the causes of unacceptable performance, not just compensate for it with band-aids and pain-killers.

What role should measures play in decision making?

Using performance measures to their full power requires, as I mentioned earlier, an orchestrated effort. This means deliberately designing the approach we take to managing with performance measures. We need a new way to think about how to use information to inform us, complement our intuition and experience, and exercise our influence over the performance results we want. The onus falls on us. If we want performance to improve, it’s up to us to do whatever we can to make that happen. We can’t wait for the conditions to be right, for the planets to line up, and for everyone else to be ready before we start work on the performance results that matter. Performance improvement is not about perfection, and never should be. But it should be about continual improvement fuelled by a passion for and dedication to excellence.

What I’m going to share now is a five-step technique to stimulate and structure a conversation about performance improvement guided by our performance measures that puts the focus firmly on fundamentally improving performance, not compensating for poor performance. This technique doesn’t end with ideas for improving performance. It closes the loop by looking for the evidence that performance does indeed improve as a consequence of implementing those ideas. The five steps of the PuMP Reaching Performance Targets framework are:

Step 8.1: Set sensible targets

Step 8.2: Prioritise the performance gaps

Step 8.3: Find the causes

Step 8.4: Choose high-leverage solutions

Step 8.5: Look for signals and check for impact

Not one of these steps is rocket science, and they’ve been around since the 1920s. Since then they’ve developed to become an important part of the total quality management movement. We don’t talk about total quality management (TQM) so much these days, but the method is still the same: What result do we want? How far away are we from that result? What’s got to change to close the gap, ideally without unintended and unwanted consequences? And when we change those things, has the gap closed as expected and without unintended and unwanted consequences? Today this basic philosophy is embedded in what we refer to as process improvement, with variations by way of methodologies such as Lean, Six Sigma, and Business Excellence. The five PuMP steps for using measures to reach performance targets are simple habits that are the foundation of almost every successful continuous performance improvement effort. If we leave one of the steps out we lose control or influence over performance improvement, and have to rely on luck or good fortune instead. That’s a very stressful way to manage an organisation or business.

The five steps of the PuMP Reaching Performance Targets framework do not prescribe a decision-making process, but rather a natural flow for a conversation that should occur throughout our decision processes and performance review meetings. As you read on and learn more about each of the five steps, ask yourself’How would I embed these steps into a formal and regular performance and strategy review process at my organisation?’

The 5 steps for using measures to reach performance targets

Step 8.1: Set sensible targets

We’ve arrived at one of the logical places in the performance measurement process to think about targets. Once we understand where the performance level truly is now using our XmR charts, we can assess how much of a performance improvement we can realistically aim to achieve. That expectation can become our first target—the short- or medium-term target everyone believes is achievable. We can also set a longer-term or stretch target that we don’t know exactly how to achieve, or whether we’ll achieve it. Stretch targets can be exciting and motivating when they’re set in the right context. But we don’t set targets for 100% (or any other value that represents perfection). We rarely have enough influence over a performance result to achieve a perfect outcome. We set targets that stretch us to find as much influence as we can to get performance results to improve. Even though we might want zero workplace accidents, and setting a target of zero is what we want to do and ethically feel compelled to do, it’s not going to help people feel empowered. Anything short of zero will feel like failure even if we achieve something amazing, such as halving the number of accidents. So we set targets in the context of continual improvement, acknowledging that perfection can’t happen overnight, next month, or possibly ever. If the average is 10 accidents per week right now, then we set a target of no more than six accidents per week (or to phrase it positively, preventing four accidents per week).

It’s important not to let the improvement projects or strategic initiatives take on a life of their own. They have to stay focused on achieving a real change in our performance measures. The idea is to give time and money only to the most elegant solutions that fix—or at least reduce the impact of—the things constraining our performance. So we need to gauge how well our solutions are working.

People generally set targets without really thinking enough. What exactly are they setting the target for? Is the target the ideal level to hit every day, week or month? Is the target the ideal level to hit by the end of the year? If we’ve absorbed the messages of Step 7 in the PuMP Blueprint we will appreciate that routine variation is part of every single measure, so comparing performance each month with a target value is a dumb comparison to make. (Yes, dumb.) But when we use XmR charts, and set targets for the central line or the natural process limits, they make much more sense. We can clearly visualise what hitting the target means in that case.

The Federal Aviation Administration (FAA) in the United States wanted to reduce the time it takes to find placements for those people who don’t pass the flight controller’s exam (referred to as ‘training failures’). The FAA formed a Measures Team headed by Steve Silvers, Employee Services Team Manager of the Central Service Center. Steve and his 17 staff members provide staffing and position management support to around 10,800 employees located in 17 states in the central U.S. One of the things they do is place training failures in jobs they are better suited for at different facilities throughout the United States. The team measured this result using Training Failure Processing Time, the average number of days it takes from the date they are notified of a person failing the exam to the date human resources is notified of that person’s new placement.

Based on the XmR chart, Training Failure Processing Time was sitting at an average of 67 days. This meant a lot of lost productivity as the people who failed the exam waited for alternative assignments, and a lot of angst for those waiting in limbo to find out whether they had a job (and where it would be). Needless to say, the team responsible for keeping those folks in limbo wanted to shorten the processing time. Management gave them a target of 21 days, but it wasn’t the kind of target where from this point forward everyone had to try and place training failures into new positions within 21 days. That would have been ludicrous, because trying harder is just another way of compensating for a process that isn’t capable of performing at the level we want. The 21-day target was set for the central line in the XmR chart for Training Failure Processing Time, as shown in Figure 41. A second target of 30 days was also set, which the team called an Interim Target—a more achievable level of improvement to aim for on the way to the ultimate target of 21 days.

Notice how the targets are represented on the chart in Figure 41 as little dots that sit just above the dates by which they should be achieved. As time progresses toward those target dates we will easily see how well the central line in the chart approaches the level of the targets. What we’d want to see is a signal or two that the average Training Failure Processing Time is dropping, and a recalculated central line that moves closer to the target dots.

FIGURE 41. Training Failure Processing Time targets
FIGURE 41. Training Failure Processing Time targets

ESTABLISH A CURRENT PERFORMANCE BASELINE

Does it go without saying that it’s better to set targets after we have established a baseline of current performance? If we do it’s so much easier to see the size of the improvement we’re aiming for, and whether it’s too aggressive or not aggressive enough. It also helps us see how setting two or three targets can help everyone feel well-paced in their improvement effort, and not freaked-out by a scary stretch target. So the first target should be a modest improvement that most people believe is achievable, with the second possibly more challenging but still one people can see is possible. Finally, the stretch target can be one that no-one is sure whether it’s possible, or how to even reach it. We can worry about how to reach that target after we achieve the first two. So target timeframes don’t have to align with our planning horizon. We don’t have to limit ourselves to an end-of-year target. We can have additional along the way, and beyond.

DEFINE THE SIGNALS TO RESPOND TO

When we understand the XmR chart signal types (Step 7 of PuMP) and how to set sensible targets, we might want to review the response section in our measure’s definition (from Step 5 of PuMP) to be more specific about the rules for interpreting each performance measure. We might recall from Step 5 the following signal types, which ought to make a bit more sense now:

Signal 1: Performance has reached or exceeded the target.

Signal 2: Performance is improving at a rate fast enough that the target will likely be met.

Signal 3: Performance is improving, but not fast enough to reach the target.

Signal 4: Performance is stable and not changing.

Signal 5: Performance is getting worse.

Signal 6: Performance is unpredictable or chaotic.

This method of target setting also has implications for how we use traffic lights—those red, amber and green symbols on dashboards that summarise whether performance is unacceptable, so-so, or good. Which colour would you assign to each signal in the list? I’d make signals 1 and 2 green lights, signals 3 and 4 amber lights, and signals 5 and 6 red lights.

If the initiatives in our strategic or operational plans are right we should see patterns in our performance measures that show performance improving when we expect it to, correlated with implementation of our planned initiatives. And if that is the signal we get, then we don’t need to go beyond this first step in responding to that performance measure. But if that’s not the signal we get, and our performance measure isn’t showing improvement despite our chosen initiatives, then something needs to be done. But we don’t do just anything. We have to find the root cause first and make the change there.

Step 8.2: Prioritise the performance gaps

Performance management is about closing the gaps between actual performance and the levels of performance we deliberately choose to pursue. It’s not about making everything better. It’s about making the most important things better—injecting energy, time and money into the fewest things that give us the best chance of fulfilling our mission and achieving our vision. Even if the nicest person on the planet says everything matters, they’re wrong. Remember the manager I mentioned back in Chapter 2 who had 87 KPIs? The stress he put his staff through, and the nervous breakdown he suffered himself, were sure signs that 87 priorities is too many. You can’t achieve performance excellence without sharp focus.

Next to having a ruthlessly small set of performance results and accompanying measures, another tenet of performance excellence is prioritising which measures needs attention first. There are a few types of signals our measures can show us when we include a target in our XmR charts. The first is the loveliest: actual performance is at or better than the target. The second is comforting: performance is moving toward target, and there is still time to reach the target by the due date. The third is frustrating: performance is improving but nowhere near enough to reach the target on time. The fourth is disturbing: despite our efforts, performance isn’t changing at all. The fifth is alarming: performance is moving in the opposite direction from our target. And the sixth is even more alarming: it shows chaotic performance, which means we have little control or influence at all over that performance result.

Each signal carries a relative sense of urgency that when combined with the relative importance of each of our performance measures, helps to prioritise which performance gaps to close first. And that’s important because often we have more to do than we have time or resources to do it, particularly if we want to do it well.

It’s easy enough to see why the FAA’s Central Service Center prioritised Training Failure Processing Time as something to improve. Looking back at the XmR chart for Training Failure Processing Time in Figure 41, we can see that the time varied chaotically between 40 and 110 days between May 2010 and April 2011. The natural process limits suggest we could expect it to take as long as 148 days. This is Signal 6: Performance is unpredictable or chaotic. Generally, the approach to take when we see unpredictable or chaotic performance is simply to examine the business process and make it consistent. Standardise the tasks in the workflow that have the greatest impact on the results the process ought to deliver. The chaos is often a sign things are happening in an ad hoc fashion.

We set targets to raise the bar, become better than we have been in the past, and focus us on a real improvement. To be better, we have to change something. The gaps between actual performance and targeted performance aren’t closed just because everyone tries harder. Almost always, something fundamental in the work process has to be changed, such as the procedures, skill levels or technical systems.

Changes rarely happen overnight, and they don’t always work as well as we intended. So it’s unreasonable to expect that next month should hit the target. There is a time lag for performance to fundamentally improve, and for us to see a signal in our performance measure that performance has improved and by how much. That time lag is because we are working out what is holding performance down and making the changes to elevate performance.

Step 8.3: Find the causes

Step 8.3 of the PuMP Reaching Performance Targets framework is finding the causes of the signals we see in our performance measures—the triggers that let us know if, when and how we need to respond. When we see a signal (if we see a signal), we want to find out what caused it before we spend any time or effort to improve performance. Cause analysis is a systematic investigation that often uses additional data to understand what was happening when and where a signal occurred. But before we go looking for the data, we want to first be sure we understand the system that is producing the result we’re measuring.

UNDERSTAND THE PROCESS PRODUCING A PERFORMANCE RESULT

In this discussion of cause analysis, I’m referring to business process management in general, as well as methods such as Six Sigma and Lean that are specifically for business process re-engineering. Rather than repeat what has been documented in a myriad of books and websites about process improvement techniques, I’ll simply discuss two techniques that are particularly useful in guiding our cause analysis.

First, flowcharting (or process mapping) helps us define the scope of the system producing the performance result we’re measuring. FAA’s Central Service Center examined the existing (or ‘as-is’) method of processing the individuals who failed air traffic control training into new placements. Valerie, one of Steve’s colleagues in the FAA Measures Team, worked with a group of field managers and the executive adviser to brainstorm the problems they were seeing from a field perspective. They brainstormed what they thought was delaying the process, and the complaints they were getting from the employees and the union. And they created a flowchart of the entire process. As you might agree after looking at Figure 42, they discovered that much of the time taken to process training failures was spent waiting: waiting for responses, waiting for approvals, and waiting for the failed air traffic control specialists to make their decisions about alternative placements.

FIGURE 42. The ‘as-is’ Training Failures process
FIGURE 42. The ‘as-is’ Training Failures process

Flowcharts of a workflow or business process, like the one in Figure 42, provide a big-picture view of where problems are and their scope. That’s the starting point for finding the root causes of the problems.

SCOPE OUT POSSIBLE CAUSES OF PERFORMANCE GAPS

With flowcharts or maps of the business process before us, we can use fishbone or Ishikawa diagrams to scope out the range of potential causes of our measures’ signals. These diagrams are useful because they give us a practical structure for thinking through relevant causes as a preliminary step to guide our data analysis. Each ‘bone’ that angles off the spine in a fishbone diagram can be a general category. Within each general category, we can brainstorm related causal factors. In manufacturing processes, a common array of categories is: Equipment, Process, People, Materials, Environment, and Management. For service processes, a good array of categories is: Price, Promotion, People, Processes, Place/Plant, Policies, Procedures, and Product. Common categories for service are: Surroundings, Suppliers, Systems, and Skills.

Figure 43 is an example of how a fishbone diagram would have captured the major causes of the large Training Failure Processing Time for the FAA Measures Team.

USE DATA ANALYSIS TO VERIFY THE ROOT CAUSES

Once we have scoped out the range of potential causes of the signals in our performance measures, we can frame questions that give us a solid starting point for data analysis.

Today we talk about ‘analytics’, but as a statistician I’ve always called it ’statistical analysis’ or ‘data analysis’. Whatever we call it, it’s a very important part of cause analysis. Business intelligence applications make it relatively easy, and most of the statistical part lies underneath the surface so we don’t need a high degree of skill to do it. It’s just a question of dragging and dropping pieces of data onto different types of charts, and we end up being able to answer all sorts of questions quite easily. Stephen Few’s visual analysis and dashboard design books⁵² are fabulous references to guide the data analysis component of cause analysis, particularly good one is Show Me the Numbers, a comprehensive, eloquently written and generously illustrated guide for the everyday businessperson to design tables and graphs that draw the story out of all kinds of data sets. Instead of jumping to premature conclusions about what’s causing the signals in our performance measure with hearsay, gut feelings and guessing, we go to the data with questions, curiosity and confidence.

FIGURE 43. fishbone diagram of causes of Training Failure Processing Time
FIGURE 43. fishbone diagram of causes of Training Failure Processing Time

Some of the most useful techniques for data-driven cause analysis are also the simplest. Pareto charts are very effective for quickly examining the degree to which a range of factors might be associated with the signal we see in our performance measure. Because the FAA Measures Team noted that waiting time was a big contributor to the total processing time for training failures, it could have been very useful to find out the main reasons for delays. This analysis could have been displayed in a Pareto chart of the percentage of waiting time accounted for by each reason, like the example in Figure 44.

Scatter plots offer another simple and insightful way to analyse data and explore the factors that correlate with performance measurement results. However, it’s critical to keep in mind that correlation is not the same as causation. A few funny examples of this warning appear on a blog for statisticians and data analysts.⁵³ One particularly cool example (ahem) is the scatter plot in Figure 45, which shows how global average temperatures have increased with the decline of the pirate population. Of course it’s ludicrous to think pirates have anything to do with global warming (and I’m not sure how reliable this data is). But there are plenty of silly leaps from correlation to causation that aren’t as obviously daft. Just because there is correlation in two sets of numbers doesn’t mean there is any sensible relationship (particularly a cause-effect relationship) between the two measures.

FIGURE 44. Example of a Pareto Chart for Training Failure Processing Time
FIGURE 44. Example of a Pareto Chart for Training Failure Processing Time

PRINCIPLES TO GUIDE ROOT CAUSE ANALYSIS

Below are some general principles to bear in mind as we look for the causes of signals in our performance measures. Adhering to these principles will help ensure we find real causes to treat instead of patching up performance with band-aids or painkillers.

Influence, not control The first principle is to be bold enough to explore our influence over performance, not just what’s within our direct control. One of the most common excuses I hear from people who don’t want to measure something is that it’s not within their control. One of my clients is in a team of geological engineers in a mining company whose job it is to find profitable new sites for mining. They can’t guarantee when, where or how often they will make such discoveries, or even whether there will even be discovery. So they didn’t want to measure their discovery rate. But they can’t say they don’t have any influence over it. That’s the whole point of their job—to design and execute the discovery process. If the company wants to make more or better discoveries, they can certainly change their discovery process. They could try different technology to help the search, use different information to assess potential discovery opportunities, or streamline the steps to allow them to evaluate more opportunities in a shorter timeframe. That’s exercising influence over the outcome, even if they don’t have control over it. And when we realise our target doesn’t have to be perfection—we’re aiming only to make continual improvements— it’s easier to see that we can indeed influence performance improvement, at least to some extent. Possibly the root of this excuse is the old mantra, ‘You cannot control what you do not measure’. I think a better mantra for performance excellence is ‘Influence, not control’.

FIGURE 45. Pirates cause global warming?
FIGURE 45. Pirates cause global warming?

Look at patterns, not points The second principle is to look for causes of the patterns or signals in a performance measure, not individual points. If we try to explain the reasons every month is different from the last one we’ll just go on a tampering escapade that will lead to a lot of wasted time, effort and money, and bring about more chaos than control. Remember the sawmill and their kneejerk reactions to their performance dashboard’s signals?

Setting targets requires understanding why performance is falling short The third principle is to accept that sometimes we can’t set a meaningful target until we have some idea of why performance isn’t currently good enough. The way we’ll reach our targets is by removing or reducing the effect of the causes on our performance result, which is going to cost time and money. If our target is too big too soon, we can demoralise people when they are held accountable for something they don’t have the resources to rectify. But when they know the causes of under-performance, they can more easily explore the influence they can have to pursue an affordable target.

Don’t look at one measure in isolation Finally, remember the principle of not just looking at one performance measure in isolation. Rather, look at the patterns among all the measures related to one another. Our Results Maps and Measure Definitions will have identified the performance measures related to one another in our organisations, and when these measures are reported together we’ll get a more complete story that focuses us on looking for root causes rather than treating symptoms.

All of the analysis Steve and his Measures Team in the FAA did to understand the causes of delays in the training failure process revealed that getting approvals was the biggest time-eater. Bureaucracy at its finest.

Step 8.4: Choose high-leverage solutions

If your experience is anything like mine, the most common response to poor performance results is to make excuses. ‘It’s outside of our control.’ ‘It’s their fault.’ ‘It’s the economy.’ Truth is, almost everything is outside of anyone’s complete control. We’d be doing ourselves a really big favour if we simply assumed everything is outside of our control. Then we’d be free to focus only on our influence. We need to have this mindset before we can fundamentally improve performance. Let’s not wait until we have complete control. Let’s not wait for ‘them’ to fix it. Let’s not wait for the economy to improve. Let’s embrace the idea that ‘if it’s going to be, it’s up to me’, and look for solutions that address the fundamental obstacles to achieving the performance we want.

Playing the blame game is not on. The healthy and emotionally intelligent way to deal with performance gaps is to find out what is holding those gaps open. This is why the design of performance reports includes cause analysis: to focus the discussion on ‘why?’ rather than ‘why not?’, and on exercising influence rather than making excuses.

FIGURE 46. The new Training Failures process
FIGURE 46. The new Training Failures process

There was too much delay built into the FAA’s ‘as-is’ process for training failures to make it worthwhile trying to improve the existing process. Instead, they redesigned their process from scratch. In doing so, they sought and won support from both the unions and management. One of the greatest changes was the directors giving a ‘blanket approval’ to the team to place training failures at facilities that were staffed below their staff number threshold. This was the case at least 90% of the time, and the effect on reducing delays in the process was significant. Compare the new process in Figure 46 with the original process in Figure 42, and you’ll see far greater simplicity in the process design.

REMEMBER THE RELENTLESS SOLUTION FOCUS

Earlier in this chapter 1 mentioned Jason Selk and his attitude of a relentless focus on solutions. A relentless solution focus is the only way we will ever achieve outstanding performance or reach our performance targets the things we say we want when we write our goals. It takes hard work and undying devotion to truly stand out and excel. And making excuses or giving up on a goal are conspicuous clues that the get-up-and-go to achieve that goal either got up and went, or was never there in the first place. Those excuses are clues that the lofty words in the strategic plan are empty platitudes.

KEEP SOLUTIONS FOCUSED ON THE ROOT CAUSES

When we start the search for solutions, it’s going to help if we keep the obstacles constraining performance the most front and centre in our view. Most solutions treat the symptoms of unacceptable performance, essentially compensating for it rather than truly fixing it. For example, a billing team who followed a similar approach to the FAA training failures team wanted to increase the percentage of invoices paid on time and in full. They first believed educating their customers about on-time bill payment was the solution. But the real solution turned out to be simplifying their pricing model and increasing the accuracy of their invoices. The question to start a search for solutions to improve performance shouldn’t be ‘How do we improve our measure?’ It should be ‘How can we reduce the impact of this particular cause of poor performance?’

A high-leverage solution has a high benefit-to-cost ratio. We get a big return for the investment. It’s like the sugar train example, where a very small investment in modelling the sugar system produced far greater capacity in the rail system to meet recent and future growth in sugar production. And this investment saved money too. The organisation was prepared to invest millions of dollars in new trains and wagons to achieve the same outcome. But the new trains and wagons would have simply been an expensive painkiller, not a true solution.

To find high-leverage solutions we have to do the opposite of the advice ‘Don’t just stand there, do something’. Often we should instead ‘Don’t just do something, stand there’. We need to stand there and look more carefully at the options we have for dealing with the causes of poor performance. We might even need to stand there long enough to test a few different options—potential solutions—and get some objective evidence about their relative impacts. This means pilot testing or simulating potential solutions to gauge their impact and cost-benefit ratio before we make a decision about what to do.

PROJECT MANAGEMENT VERSUS PERFORMANCE MANAGEMENT

If we want to properly manage the implementation of a performance improvement, we need to rely on proper project management strategies. Part of proper project management is to use milestones to pace the successful progress of the project over time. At this point, it’s worth remembering our observation from Step 3 of PuMP that many people still use milestones as performance measures. I argue that milestones are different from performance measures, and now it’s easy to see why. A milestone is a project management tool that helps us keep track of the project’s implementation. But it doesn’t give us any indication about the degree of success of the project in achieving its intended outcome (such as its impact on a performance result). That’s what true performance measures do: measure the degree of success of something in achieving the desired impact. Milestones and performance measures are completely different from one another. Both are important, but they serve different purposes.

Step 8.5: Look for signals and check for impact

Almost identical to Step 7.5 of the PuMP framework for interpreting signals from measures is this final step in the framework for reaching performance targets: look for signals during and after the implementation of solutions to fix the causes of poor performance. It might feel like the job is done when the improvement project reaches its final milestone, and everyone will be keen to rush off to the next exciting project. But it’s essential to continue checking the measure for signals that confirm whether or not the fix we implemented really did some fixing.

After implementing their process improvements, FAA’s training failures team saw the impacts in their measure’s XmR chart (Figure 47). You can imagine how pleased everyone was when the chart showed they not only reached their interim target of 30 days ahead of schedule, but also exceeded their original target of 21 days. Training Failure Processing Time dropped down to 20 days as a result of their process redesign, and it’s now far more predictable and in control. They went ahead and set the next target at 14 days. Do you think they could have set a 14-day target and believed it was achievable back when it was taking an average of 67 days to find placements for people.

FIGURE 47. Improvements in Training Failure Processing Time
FIGURE 47. Improvements in Training Failure Processing Time

The FAA team’s great work in reducing the Training Failure Processing Time reduced lost productivity and saved nearly half a million dollars over the seven-month period the improvements were made. And of course, the savings will continue as long as the Training Failure Processing Time is kept under tight control.

ISOLATING THE IMPACT OF OUR INITIATIVES ON PERFORMANCE MEASURES THAT ARE AFFECTED BY OTHER FACTORS BEYOND OUR INFLUENCE

It won’t always be easy to know whether or how much of our solution is the reason performance improves, particularly when other people, organisations or processes also influence that performance measure. We can do a few different things to isolate the effect of our own solutions or strategies, although they do go against the grain and momentum of several entrenched business practices.

Here’s an example. Owners of a data intelligence company that provides expert market analysis for its clients wanted to improve the success of their promotion activities. They wanted to find ways to increase their Sales Conversion Rate, the percentage of prospects who convert to customers as a result of promotional activities. Francois, one of their lead analysts, suggested using control groups to provide a baseline for comparing the relative impact of each improvement idea. Ironically, the company decided against using control groups even though doing so is a proven way to measure the isolated impact of a change. Why? Because using control groups would reduce the commissions of salespeople. They concluded that because those in the control groups would not be exposed to the new promotion approaches that were being tested, and therefore less likely to become customers, they would be less likely to produce commissions for the salespeople working with the control group members.

Business experiments⁵⁴ are very important for testing the true impact of investments in performance improvement. There are two primary methods for designing business experiments to objectively gauge the impact of solutions on performance. The first is the use of control groups, as was proposed in the example of the data intelligence company. If we want to improve the leadership style of managers throughout an organisation (perhaps to increase the Employee Engagement Score) we might halve the group of managers randomly (so we don’t introduce any bias), provide leadership training for one group and do nothing with the other group. The second group is our control group, which makes it possible to monitor whether the training increased the Engagement Score for employees of managers in the first group compared to employees of managers in the control group. The economy can also influence employees’ engagement, as can their relationships with colleagues or how much they like their job. But by doing an experiment with a control group, we isolate the effect of the one change we want to try because both groups are still influenced by the economy and employee relationships. In other words, the only major thing that differs between the two groups is the one idea we are testing to see whether it helps us exercise more influence over our performance result. We’d use two XmR charts for our performance measure in this case—one for the control group’s performance, and one for the experiment or treatment group. Ideally we’re looking for a signal there has been a good change in the central line for the group that got the leadership training. The difference between the central line values for the training group and the control group is the size of the impact of our initiative. (Of course, we would make sure the managers in the control group were not penalised in any way if they exhibited weaker leadership performance than the managers who received leadership training. Remember, performance measures should not be used to evaluate people because it stifles true performance improvement.)

A second way to gauge an investment’s impact on performance results that lie outside our complete control is to use simple time-series analysis. We note the time we implemented the solution, and then look for signals in the XmR chart for that performance measure. If the solution takes a while to bed down, then we’re looking for a signal that might take months to appear in the chart. If it’s a solution we can implement overnight, we’re looking for a short-run signal.

On finding a valid signal in our XmR chart, we’ll recalculate the central line and natural process limits. The size of the change in the central line compared with what it was before the initiative will confirm the size of impact of the chosen initiative. It’s not always easy to truly isolate the effect of a strategy, but that’s no excuse for not trying.

Evidence-based management

The habits I’ve described above should ideally be the way we proceed during planning, before strategic initiatives are chosen and a budget is allocated. By using our performance measures as part of our strategic analysis we can be confident about the success we will achieve by investing the budget for an initiative. But these habits should also continue throughout the planning horizon to be sure the initiatives are working as expected. At the very least we want to incorporate these habits into our performance review conversations. Essentially, they’re the habits of evidence-based management. When results are claimed, successes declared, or hypotheses taken as givens without any sufficiently supporting information it’s time to test and probe. If we can’t find any data, facts or evidence, we have two choices: bear the cost of gathering the requisite data, facts or evidence, or bear the cost of making the wrong decision. Only one of these is the responsible choice to make.

Rest assured, challenging the dialogue of the decision process is going to take us and our colleagues into some uncomfortable territory. Egos feel poked at, fears have to be faced, and more effort has to be expended on planning, implementing and measuring processes than some might have hoped or imagined. But it is a job that must be done. You’re the one reading this book, so there’s a good chance it’s a job that must be done by you. ‘They’ most likely won’t do it.

Take Action: Redesign your performance review forum

To adopt the new habits described by the PuMP Reaching Performance Targets framework, we need a forum where owners of the performance measures can have a dialogue about the signals in the measures, the likely root causes, and the potential solutions that will improve performance. These forums often take the form of monthly performance review meetings, as was the case for the senior executive team when I worked at Queensland Rail. But they can also be weekly team performance meetings, like the Toolbox meetings that various teams at the railway workshops in Queensland Rail held when they began adopting Six Sigma as their improvement methodology. Annual performance review meetings are too infrequent to nurture a continuous performance improvement philosophy.

So review your current performance management process, and make sure the frequency and the agenda of meetings are designed to facilitate the evidence-based approach of the PuMP Reaching Performance Targets framework.

Chapter summary

Measures should never be used to judge people. We will never have a true and strong performance culture if people’s behaviour is influenced by a constant shadow of judgement cast over them. They won’t be nearly as collaborative, won’t be completely honest about the data that feeds the measures, and won’t have their attention on the greater good. Performance measures should be tools people use to work on the business collaboratively and openly to improve it.

We need to learn and practice a continuous improvement philosophy for managing performance. Our measures are pivotal to this. But we still need to have a disciplined, deliberate approach to improvement that includes cause analysis, a relentless solution focus, and testing to ensure that solutions we’ve spent time and money on have actually worked.

Establishing routine performance review meetings with performance measures at their heart is key to true performance improvement. We need to embed the steps of interpreting performance signals, investigating the root-causes, and designing and evaluating the impact of solutions into routine performance review meetings.

Your Measures Team has come a long way and achieved a great deal by the time they complete PuMP Blueprint Step 8—the final step. But rather than let the momentum wane, build it so you can continue to build your organisation’s or company’s performance culture.

12

Resources for PuMP implementation success

The idea of using a pilot approach to implement PuMP is to learn how to measure performance well, a lot sooner, and with much less effort and angst than trying to do it wholesale from the outset. There’s plenty to unlearn about performance measurement in addition to learning new techniques. And on top of that we face people’s prejudices and fears about measuring performance, which can cause them to disengage. A pilot approach gives us permission to learn, unlearn, and engage while making real progress on measuring something that matters.

We can’t set the goal of building our company’s or organisation’s entire performance measurement framework in one fell swoop. If we do we’re doomed to fail, which will fuel (more) resentment, more bad performance measures, and more time wasted that otherwise could be spent making performance better. Instead we need to start small and let the energy, enthusiasm and buy-in grow on the back of small wins.

My intention with this book is that we learn (and unlearn) enough that we can start to play with PuMP on a small scale. Enough to get a taste for how it works, and for good measures and real performance improvement. Enough to decide whether PuMP will ultimately be the basis for building our company’s or organisation’s entire performance measurement framework.

This chapter presents a range of additional resources that can help us play with PuMP and get the small wins we all want.

Download templates from the book’s website

www.pumpblueprintbook.com

Throughout the book I’ve mentioned resources that can help us plan our PuMP Blueprint implementations. They are free to use within your own organisations or businesses, but not to use commercially. If you’re interested in becoming a licensed PuMP Consultant, contact us at info@staceybarr.com.

You can download the templates mentioned throughout this book at the book’s website (www.pumpblueprintbook.com):

PuMP Pilot project plan template

PuMP Mind Map exercise templates

PuMP Diagnostic Discussion Tool

PuMP Strategic Plan template

PuMP Results Map template

PuMP Measure Design template

PuMP Measure Gallery instruction checklist

PuMP Measure Definition template

PuMP Performance Report document and spread sheet templates

XmR basic instructions and spreadsheet template

All you need is the password: pumpmykpis

Join the ‘Measure Up’ community

www.measureupblog.com

Would you like to stay in touch? I publish a regular free email newsletter called Measure Up, which contains brief updates from me, practical tips for performance measurement, and links to upcoming events and other articles, resources and people who can contribute wonderful things to improving our performance management. I also publish practical tips on the Measure Up blog, where you can comment, ask questions, and participate in discussions.

You can also join the Measure Up LinkedIn Group and participate in discussions about all things performance measurement. Visit:

http://www.linkedin.com/groups/Measure-Up-4419275?gid=4419275

Add to your performance measurement library

I wish I loved reading as much as I love books. Sometimes I meander aimlessly around a book and pluck ideas from random pages. Sometimes I procrastinate for months before reading beyond the first chapter. And sometimes I devour a book like a block of dark, 70%-cocoa chocolate (i.e. in a matter of days). The books I recommend to supplement or shape your performance measurement knowledge fall into the dark-chocolate category. They are the books that have most shaped my own views and approaches to performance measurement.

Transforming Performance Measurement: Rethinking the Way We Measure and Drive Organizational Success by Dean R. Spitzer, Ph.D. (AMACOM, 2007)

Death Sentence: The Decay of Public Language by Don Watson (Random House Australia, 2003)

How to Measure Anything: Finding the Value of ‘Intangibles’ in Business by Douglas W. Hubbard (John Wiley & Sons, Inc., 2007)

Open Space Technology: A User’s Guide by Harrison Owen (Berrett-Koehler Publishers, Inc., 1997)

The Non-Designer’s Design Book by Robin Williams (Peachpit Press, 1994)

Show Me the Numbers by Stephen Few (Analytics Press, 2004)

Information Dashboard Design (2nd Edition) by Stephen Few (O’Reilly Media, Inc., 2006)

Understanding Variation: The Key to Managing Chaos by Donald Wheeler (SPC Press, 1993)

A checklist to plan your PuMP implementation

Most performance measurement systems are never fully brought to life because of poor implementation, planning and execution. The following prompts are a framework to design our performance measurement system, acknowledging that it is a change process just like any other initiative our organisation undertakes in the spirit of continuous improvement and adaptation. We can refer to this checklist as we plan our PuMP Pilot, using the project plan template from the book’s website.

DESCRIBE THE DIFFERENCE YOUR PERFORMANCE MEASURES WILL MAKE.

❏ What will it mean to have performance measurement working well in your organisation?

❏ What is the greatest benefit great performance measures can bring? Goals are achieved faster? More goals are achieved?

CHECK WHO HAS CONTROL OVER INITIATING AND MAINTAINING THIS DIFFERENCE.

❏ Where do you have influence within the process of developing and using performance measures in your organisation?

❏ Whose leadership, help and commitment will be important to complement the influence you have?

❏ How will you speak with, inspire and influence these people?

DESCRIBE THE DIFFERENCES IN RICH SENSORY DETAIL.

❏ How do you want people to respond to developing and using performance measures?

❏ What will people be doing when they have great performance measures?

❏ How will your organisation be different when it has and uses great performance measures?

❏ What old artefacts will be gone, and what new artefacts will replace them?

REFLECT ON WHY YOU WANT PERFORMANCE MEASURES.

❏ What is the driving reason that you want performance measures, or that your organisation should have them (or improve them) now?

❏ What can’t your organisation achieve right now without performance measures?

❏ Will performance measures really be an important way to achieve these things? What else will you need to do?

DEFINE THE EVIDENCE THAT WILL LET YOU KNOW PERFORMANCE MEASURES ARE MAKING THE DIFFERENCE THEY SHOULD.

❏ What are the signs of the kind of performance measurement culture you want to nurture and mature?

❏ What will convince you and others that you have the ‘right’ measures?

❏ How will you know people are producing and using the measures properly?

❏ What are the indicators or fags of unwanted unintended consequences, or performance measurement ‘going bad’?

EXPLORE HOW PERFORMANCE MEASURES WILL AFFECT OTHER THINGS AND BE AFFECTED BY OTHER THINGS.

❏ How are people likely to respond to the changes that come with developing and using performance measures?

❏ Which organisational systems, processes or structures help or hinder development and use of performance measures?

❏ For each effect you ponder, how can you avoid, overcome, work around, work with or compensate for it?

ARTICULATE THE PRINCIPLES THAT WILL GUIDE THE CHANGE YOU ARE TRYING TO MAKE.

❏ What behaviours will be important to role model as you develop measures for your organisation?

❏ What are the different ways people throughout the organisation can be involved in the process in an appropriate way for them?

❏ What philosophy or principles of performance measurement will be important to weave through everything you do?

PLAN WHAT NEEDS TO HAPPEN—WHEN, WHERE, AND WHO’LL BE INVOLVED.

❏ How will you engage the right people at the right times, stimulating their awareness, desire, knowledge and action?

❏ How will a framework that links measures to all levels of planning and decision making in the organisation be designed?

❏ How will those measures be brought to life so they are regularly reported to the right people at the right times?

❏ How will you ensure the measures will continue to be used, reviewed, and replaced when no longer relevant?

WORK OUT THE RESOURCES YOU WILL NEED TO GET FROM ‘PLAN’ TO ‘REALITY’.

❏ What amounts and types of funding, time, technology, space and knowledge do you need?

❏ Where will these resources come from?

❏ What will have to stop, be delayed, or be changed to make these resources available?

Earn PuMP Certification

A book can give us the ideas, and a training program can help us explore those ideas deeply so we can readily put them into action. The following training programs have been around far longer than this book, and my own learning from all the lovely people who have attended them has most certainly made this book far better than it would have been otherwise.

The training programs cover essentially the same content as this book. However, the dynamic and responsive nature of live training means these programs cover examples and ideas not included in this book. Likewise, there will be content in this book that is not necessarily covered in the training programs. As well as consolidating and ‘bringing to life’ the concepts in this book, a training program gives us the guidance to apply PuMP to develop our own measures and benefit from the feedback and advice of the PuMP trainers.

Both the PuMP Performance Measure Blueprint Workshop and the PuMP Performance Measure Blueprint Online Program are certified by the Balanced Scorecard Institute and the George Washington University College of Professional Studies. Take the training, and then take an online exam to earn your PuMP Certification.

The PuMP Performance Measure Blueprint Workshop

www.performancemeasureblueprint.com

The PuMP Blueprint Workshop, or PMBW as we call it (because it’s a compulsion to turn any long name into an acronym), is a live, interactive two-day learning process where we focus on each of the eight PuMP Blueprint steps in turn. We provide you with a comprehensive workbook with examples and checklists, templates for each PuMP technique, hands-on activity time to practice applying them and get a little one-on-one coaching, and time for questions and group discussion. This workshop is available publically around the world, and also available in-house for your organisation or company.

The workshop is designed for strategy and planning officers and managers, corporate performance officers, business analysts, Measures Teams, and managers wanting to understand the practicalities of evidence-based management.

You walk away with the completed blueprint for at least one goal you want to measure more meaningfully, and the blueprint for measuring anything else that matters too.

The PuMP Performance Measure Blueprint Online Program

www.performancemeasureblueprintonline.com

The PMBO (again, the compulsion to make an acronym) is a self-paced online learning program that follows the curriculum of the PuMP Blueprint Workshop. We focus on one step of the PuMP Blueprint at a time in a 90-minute lesson for each one, which you access through the PMBO members-only website.

The website has all the lesson videos, examples, workbooks, templates and activity instructions you need to manage not only your own learning, but also your implementation of the PuMP Blueprint in your business or organisation.

The PuMP Professional Program

www.pumpprofessionalprogram.com

If leading performance measurement implementations is a regular part of your job, you can learn how to increase your impact and save loads of time through the PuMP Professional Program. It’s an online coaching program where we take you to the next level of mastery with the PuMP Blueprint: how to facilitate others to understand and implement PuMP with confidence and ease, and how to systemically build a corporate-wide performance measurement framework.

With even more resources available on the program’s website, you’ll save time preparing and delivering PuMP implementations, and rapidly transform the performance of your organisation or business. That’s what makes the PuMP Professional Program perfect for strategy and performance professionals whose role includes facilitating development of measures within their organisations or companies.

A prerequisite for the PuMP Professional Program is completion of either the PuMP Blueprint Workshop or the PuMP Blueprint Online Program.

Notes

What Exactly Is Performance Measurement?

1. Don Watson. Death Sentence: The Decay of Public Language. Random House Australia, 2003.

2. Dermot McKeon. Measuring Your Media Profile. Gower, 1995.

3. It is probably obvious to you that I made this up in its entirety. What is probably not as obvious to you is that I am a chocoholic, but with no intention to reach out for help.

4. Howard Rohm, David Wilsey, Gail Stout Perry, Dan Montgomery. The Institute Way: Simplify Strategic Planning and Management with the Balanced Scorecard. The Institute Press, 2013.

5. Andy Neely, Chris Adams, Mike Kennerley. The Performance Prism: The Scorecard for Measuring and Managing Business Success. Financial Times Prentice Hall, 2002.

Does Your Performance Measurement Process Work?

6. Stephen Few. Information Dashboard Design: Displaying data for at-a-glance monitoring (2nd Edition), Analytics Press, 2013.

Step 1: Understanding Measurement’s Purpose

7. Dan Pink. Drive: The Surprising Truth About What Motivates Us. Riverhead Books, 2009.

8. Don Watson. Death Sentence: The Decay of Public Language. Random House Australia Pty Ltd., 2003.

9. Douglas W. Hubbard. How to Measure Anything: Finding the Value of ‘Intangibles’ in Business. John Wiley & Sons, Inc., 2007.

10. Dean R. Spitzer, Ph.D. Transforming Performance Measurement: Rethinking the Way We Measure and Drive Organizational Success. AMACOM, 2007.

11. Stephen Few. Information Dashboard Design: The Effective Visual Communication of Data. O’Reilly, 2006.

12. A. Larry Elliot & Richard J. Schroth. How Companies Lie: Why Enron is Just the Tip of the Iceberg. Nicholas Brealey Publishing, 2002.

13. Donald J. Wheeler, Ph.D. Twenty Things You Need to Know. SPC Press, 2009.

14. James P. Womack & Daniel T. Jones. Lean Thinking: Banish Waste and Create Wealth in Your Corporation. Simon & Schuster UK Ltd, 2003.

Step 2: Mapping Measurable Results

15. http://en.wikipedia.org/wiki/Weasel_word, accessed 18th April 2013

16. Don Watson. Death Sentence: The Decay of Public Language. Random House Australia, 2003.

17. Don Watson. Watson’s Dictionary of Weasel Words. Random House Australia, 2005.

18. In the second of the three Austin Powers movies The Spy Who Shagged Me, Austin Powers’ nemesis, Dr. Evil, is cloned, and his clone is a tiny version of himself named Mini-Me. Played by actor Verne Troyer, Mini-Me always reminds me of what many people do with their corporate scorecards: trying to get every department within the organisation to write goals that clone the corporate-level goals—even if the corporate goals have little to do with the individual department’s function or goals. (It even happens when having that department following its own set of priorities would better support the overall organisation’s functioning.)

19. Howard Rohm, David Wilsey, Gail Stout Perry, Dan Montgomery. The Institute Way: Simplify Strategic Planning and Management with the Balanced Scorecard. The Institute Press, 2013.

20. Phil Jones. Communicating Strategy. Gower, 2008.

Step 3: Designing Meaningful Measures

21. The ‘Strategic Directions’ described in the plan seemed to serve the same purpose as critical success factors, describing high-level priorities or themes around which goals and objectives would be developed.

22. Learn more about Gallup’s Employee Engagement Ratio at http://www.gallup.com/consulting/121535/Employee-Engagement-Overview-Brochure.aspx

23. Alex Osborn. Applied Imagination: Principles and Procedures of Creative Problem-Solving (Third Revised Edition). Charles Scribner’s Sons, 1963.

24. CAPS Research is a global research centre for supply chain management. You can learn more at http://www.capsresearch.org/

25. The measures they designed for this result were % Jobs Ordered Off Approved Contracts, % Jobs Attempted to be Ordered Off Approved Contracts, and % Jobs Where Approved Contract Was Available.

26. Dean Spitzer. Transforming Performance Measurement. AMACOM, 2007.

27. Technically this is a multi-focused result, where ‘reduce costs’ could be treated as one result and ‘eliminate waste’ as another. This is a good time to remember that practicality is more important than perfection. We can make very significant improvements in our performance measurement before we become masters of PuMP implementation.

28. Steven Covey. The Seven Habits of Highly Effective People. Touchstone, 2003.

Step 4: Building Buy-in to Measures

29. Harrison Owen. Open Space Technology: A User’s Guide. Berrett-Koehler Publishers, Inc., 1997.

Step 5: Implementing Measures

30. When children in the Victorian era asked ‘Where do babies come from?’ the answer reportedly given was ‘the stork’, to avoid explaining unsettling details.

31. To calculate a moving or rolling average, we pick a timeframe that we want our average to roll or move through, say 12 months. Then, each month’s value is calculated as the average of the previous 12 months’ points. So if we want a rolling average of expenditure, the figure we chart for February 2011 is the average of all the monthly expenditure totals from March 2010 to February 2011.

32. Isn’t this the real job of a manager—to support their team in focusing on what matters, and to continually improve systems and processes to achieve what matters?

33. KPI Library is at http://www.kpilibrary.com

Step 6: Reporting Performance Measures

34. Edward Tufte. The Visual Display of Quantitative Information. Graphics Press, 1983.

35. Robin Williams. The Non-Designer’s Design Book. Peachpit Press, 1994.

36. Find Stephen Few and his articles and books at http://www.perceptualedge.com

37. Stephen Few. Show Me the Numbers. Analytics Press, 2004.

38. Stephen Few. Information Dashboard Design (2nd Edition). O’Reilly Media, Inc., 2006.

39. Stephen Few. Now You See It. Analytics Press, 2009.

40. Tableau Software can be found at http://www.tableausoftware.com

Step 7: Interpreting Signals from Measures

41. Wilks was paraphrasing Herbert G. Wells from his book Mankind in the Making (1903). Wells’ original words were ‘.  .  .  and the time may not be very remote when it will be understood that for complete initiation as an efficient citizen .  .  . , it is as necessary to be able to compute, to think in averages and maxima and minima, as it is now to be able to read and write’.

42. http://en.wikipedia.org/wiki/Walter_A._Shewhart

43. You can find more information about this online course at www.usingsmartcharts.com

44. The only exceptions would be when we’re measuring rare events or small counts (such as severe workplace accidents, unusual discoveries, innovations, or oil spills), measures with true seasonal or cyclical patterns, or measures with a consistent underlying linear trend. We can use XmR charts for these measures, but only after we ‘normalise’ them using the appropriate techniques. These techniques, along with a lot more detailed information about XmR charts, are the focus of my online course ‘Using Smart Charts’ at www.usingsmartcharts.com.

45. Donald Wheeler, author of Understanding Variation, advises eight points as the minimum.

46. You might remember from high school mathematics that the probably of two independent events both occurring is equal to the product of their individual probabilities. So if two events each have a 50% probability of occurring, then the probability they both occur is 50% of 50%, which is 25%.

Step 8: Reaching Performance Targets

47. In Australia, when we think something is fantastic and wonderful we say it’s the duck’s guts. Try it out some time; it’s fun.

48. Jason Selk. 10 Minute Toughness. McGraw-Hill, 2008.

49. Phil Jones. Communicating Strategy. Gower, 2008.

50. AIM Employment Engagement Survey 2010.

51. Planimate software can be found at http://www.interdynamics.com/products/planimate.html

52. You can find out much more about Stephen Few, his books, and his advice at http://www.perceptualedge.com

53. You can find the blog post at http://stats.stackexchange.com/questions/36/examples-to-teach-correlation-does-not-mean-causation

54. Thomas H. Davenport. “How to Design Smart Business Experiments.” Harvard Business Review, Vol. 87, No. 2. (February 2009), pp. 68–76.

INDEX

abnormal variation, 284

accuracy of performance measures, 13

action learning, 177

action orientation, 104–6

actions

goals as results vs., 117

goals written as, 105–6

actual reliability, 40

actual-with-desired comparisons, 13, 40–43, 70, 104, 221. See also performance gaps

alignment (visual design principle), 248

annual reports, 84. See also performance reports and dashboards

Aral, Sinan, 35

Austin Powers movies, 111

average

moving, 278–79, 279f, 283

rolling, 278–79, 279f

running, 278–79, 279f

Average Handling Time (AHT) performance measure (call centres), 41, 80, 150, 231

Balanced Scorecard, 29–30

Balanced Scorecard Strategy Maps, 129–30

bar charts, 259t, 260, 260f

baseline performance, 282

establishing a current, 319

use of a non-zero, 263f

benchmarking, industry

adopting measures from, 151–54

advantages, 152

disadvantages, 153–54

types of, 151–52

blame, 42

brainstorming performance measures, 82–83, 146–49

leads to meaningless measures, 36–37

budget, 9

businesses. See organisation(s)

buy-in

Measure Galleries used to broaden, 183–85

to measures, building, 25, 66–67, 175–96

call centres, measures used in, 41, 80, 150, 231

cause analysis, 254, 266, 267f, 322, 324–25, 329

principles to guide, 326

influence, not control, 326–27

look at patterns, not points, 327

not looking at one measure in isolation, 328

setting targets requires understanding why performance is falling short, 328

See also causes

cause-effect relationships, 119, 123, 127–29, 131–33, 132f. See also cause analysis; fishbone diagrams

cause-effect thinking, 115

causes

data analysis used to verify root, 324–26

finding the, 322–28

keeping solutions focused on root, 330

treating symptoms vs., 313–15

central line (XmR charts), 39

change

over time, ability to show, 15

performance measures’ sensitivity to, 14

Chaplin, Stewart, 99

chief executive officers (CEOs), support from, 69, 73

choices, adding vs. taking away, 182–83

Churchill, Winston, 100

coefficient of determination. See R² statistic

commercialism, 122t

Communicating Strategy (Jones), 133–34, 311

community participation measures, 164, 165t, 167, 168t–172t, 169

companies. See organisation(s)

companion relationships, 123, 127, 129, 133

comparison

a performance measure as a method of, 13

of this month to last month, 85–86

comparison type(s)

appropriate presentation methods for different, 221–22, 223t

appropriate to the measure, defining the, 221–22, 223t

See also trend-over-time comparisons

comparisons, graphs used for, 258–60. See also graph

conflict relationships, 123–24, 127, 129, 133

Conrad, Joseph, 139

Consignment Service Failure Rate performance measure (freight management), 213, 215t, 234

data items for, 233t

description and definition, 213t, 234, 235–37t

formula for and variables in calculating, 216–19, 220t

intent of, 213t

notes for, 234t

owners of, 233t

presentation, 223t

Consignment Service Failure Rate trends, responses to, 230t

consultants

outsourcing the selection of measures to, 156–58

PuMP, 70

contrast (visual design principle), 248

Corporate Performance Measure Definition Dictionary, 238–39

corporations. See organisation(s)

Cost per Contact measure, 151

cover your arse (CYA), using reports to, 84–85

Covey, Steven, 169–70

creative energy, 181

critical success factor (CSF), 97

cultural capital, 122t

customer feedback, 60

customer loyalty, 162–64

customer satisfaction, 13, 14, 60, 121t, 212

customer surveys, 9

CYA (cover your arse), using reports to, 84–85

dashboards. See performance reports and dashboards

data, 218–19

validity, 200–202

data analysis, used to verify root causes, 324–26

data-ink ratio, 268

data owner, 232–33

data referencing model, 219

data sources vs. performance measures, 9–10

defining performance measures, steps to, 211–12

Step 5.1: description, 211–13

Step 5.2: intent, 213

Step 5.3: where it fits, 214

Step 5.4: calculation, 214–20

Step 5.5: presentation, 220–22

Step 5.6: response, 222–29

Step 5.7: owners, 229–33

Step 5.8: notes, 233–34

definition owner, 231–32

Deming, W. Edwards, 35, 80, 303

Designing Measures technique. See PuMP Designing Measures technique

Designing Performance Reports technique. See PuMP Designing Performance Reports technique

determination, coefficient of. See R² statistic

Diagnostic, 88–93

dials (dashboards), 38, 242, 247

in the PuMP Diagnostic, 92

digital capability, 122t

Disney, Walt, 53

dot charts, 259t

Duhigg, Charles, 79

Elliot, A. Larry, 84

Emerson, Ralph Waldo, 53

Employee Engagement, 201

Employee Engagement Ratio, 149

Employee Engagement Score, 333

engagement

of Measures Team, 94

of stakeholders, 66–67

See also community participation measures; Employee Engagement; PuMP Designing Performance Reports technique; staff engagement

events vs. performance measures, 7–8

evidence, objective, 13–14

evidence-based management, 334

excuses, 42, 309–11

executive support, need for, 69, 73

expert facilitator, need for an, 70

experts, outsourcing the selection of measures to, 156–58

facilitators. See expert facilitator

failure rate. See Consignment Service Failure Rate performance measure

Few, Stephen, 38, 84, 241, 248, 324

fishbone diagrams, 324, 325f

formulas for measures, 216–17

frequency of measuring, 216–17

fudging figures

vs. improving performance, 28

using measures to monitor people leads to, 41

gauges. See dials

goals (and objectives), 97–98

alignment with overall strategy, 119–20

helping team articulate their goals instead of dictating them, 179

measurability (see under strategy)

multi-focus, 106–8, 118

as results or actions, 117

that give insufficient focus for measurement or execution, 103

that matter enough to measure, 76

that read too much like business as usual, 108–10

weasel words and, 82 (see also weasel words)

written as actions, lacking a clear indication of intended results, 105–6

granularity of performance measures, 13–14

graph, choosing the best, 222, 254, 256, 259t, 259–60

principles to guide the process of answering the question, 257, 258f

highlighting comparisons, 258–60

maintaining integrity, 262–63, 263f

optimising simplicity, 260–61, 262f

visual performance, 263–67, 265f

gridlines, 262f

handling time. See Average Handling Time (AHT) performance measure

holding the space, 61, 66, 178

Hubbard, Douglas W., 82

improvement process, 17–18

initiatives, 99

Interpreting Measures, 26

interpreting signals from performance measures, 26, 68, 273, 301–2

most common methods of, 274–79

not reacting to noise, 274–84

‘patterns not points; signals, not noise,’ 301

Step 7.1: Measure frequently enough, 287–88

Step 7.2: Gather enough context, 288–89

Step 7.3: Use a time-series graph, 289

Step 7.4: Filter out noise, 289–94

Step 7.5: Look for signals, 295–301

Ishikawa diagrams. See fishbone diagrams

Jones, Daniel T., 86

Jones, Phil, 133–34, 311

judging

vs. collaborating to improve measures, 180

vs. learning, 311–13

using measures for, 80–81

key performance indicators. See KPIs

key result area (KRA), 97

key result indicator (KRI), 19

Koomey, Jon, 197

Kotter, John P., 175

KPI Library, 239

KPIs (key performance indicators), 1, 3, 19, 28, 98, 140

brainstorming, 82–83

definitions, 4, 19

examples of, 3–4

terminology, 4, 19

See also performance measure(s)

leaders

supporting PuMP implementation, 76

See also executive support; expert facilitator

learning

action, 177

vs. judging, 311–13

legacy measures, 149–51

line charts, 259t, 260, 261f, 266f, 290f

long run (XmR charts), 298, 299t

lost time injuries, 111

Lost Time Injury Frequency Rate (LTIFR), 113

Mackellar, Dorothea, 139

management support, need for, 73. See also executive support

mapping measurable results, 24, 65–66, 95–137. See also PuMP Mind Map exercise; Results Map

market acceptance, 11

meaningful measure design, 24–25, 66, 90–91, 139–74

steps of (see under PuMP Designing Measures technique)

meaningful strategies, how to make, 115–16

meaningless measures

brainstorming as leading to, 36–37

See also meaningful measure design

Measurability Tests, 117–19

measurable results

mapping, 24, 65–66, 95–137

See also under strategy(ies)

Measure Definitions, 239. See also performance measure definition dictionary(ies)

Measure Design technique. See community participation measures; PuMP Designing Measures technique; PuMP Measure Design template

Measure Galleries, 195

signs of having the right people at, 192

steps for designing successful

Step 4.1: Who needs to buy in?, 185–86

Step 4.2: Invite; don’t mandate, 186–88

Step 4.3: Space for dialogue, 188–89, 190f

Step 4.4: The right people will come, 189, 191–93

Step 4.5: Gather feedback and use it, 193–95, 194f

used to broaden buy-in, 183–85

‘Measure Up’ community, 338–39

Measures Team, 24, 25, 60, 88

avoiding trampling their creative energy, 181

engaging, 94 (see also staff engagement)

forming, 60–64

have a good mix of members, 62

invite (don’t mandate) people to join, 62–63

it’s about measuring processes, not people, 63–64

manager must actively participate, 63

optimal size, 60–61

your role is to be leader and facilitator, 61–62

steps to warm up

Step 1.1: Discuss the diagnostic criteria, 88–89

Step 1.2: Capture and summarise the ratings, 89

Step 1.3: Discuss how PuMP will lift the ratings, 93

media profile, 11

milestones, 99

milestones vs. performance measures, 7–8

Mind Map exercise. See PuMP Mind Map exercise

Mini-Me syndrome, 111–13, 115

mission statement, 97

moving averages, 278–79, 279f, 283

natural process limits (XmR charts), 39

noise. See under interpreting signals from performance measures; signals

numeric benchmarking, 152

objective evidence, performance measures and a base of, 13–14

objectives. See goals (and objectives)

on-time running, 80, 110

Open Space Technology, principles of, 192

organisational priorities, performance measures and, 14–15

organisation(s)

applicability of PuMP Blueprint to particular, 72

ideal future state of, 97

lacking strategy, goals, and objectives, 73–74

purpose, 97

signs that they will take performance measurement seriously, 72–73

See also specific topics

Orwell, George, 95

outliers (XmR charts), 298, 299t

Pareto charts, 256, 259t, 266–67, 267f, 326f

performance

fudging figures vs. improving, 28

impact of performance measures on, 93

See also specific topics

performance gaps

prioritising, 320–22

scoping out possible causes of, 324

See also actual-with-desired comparisons

performance management analyses, assumptions underlying traditional, 279, 284

Assumption 1: In ‘same month’ comparisons, last year was normal, 280

Assumption 2: If trend line can be calculated, there must be a trend, 280–81

Assumption 3: The world starts anew July 1 of every year, 281–82

Assumption 4: The only probable cause for a difference is that something changed, 282–83

Assumption 5: All changes happen smoothly, 283

performance management overhaul, getting support for your, 51

Performance Measure Blueprint. See PuMP Performance Measure Blueprint

performance measure definition dictionary(ies), 238–39

building your, 239–40

performance measurement, 3, 32–33

books on, 339

common struggles with, 36

ad hoc reporting leads to waste and duplication, 37–38

brainstorming leads to meaningless measures, 36–37

focusing on our circle of control leads to excuses and blame, 42

not prioritising leads to too many measures, 36

traditional reporting leads to misreading signals in measures, 38–41

using measures to monitor people leads to fudging the figures, 41

‘is everyone’s job’, 83

nature of, 31–32

obstacles to excellent, 53

problems with, 47–50

purpose of, 1, 24

understanding the, 24, 65, 79–94

reasons people want to avoid, 176–77

See also specific topics

performance measurement campaign, 75

people insisting they cannot make time for, 75

where to start, 70

See also PuMP Pilots; specific topics

performance measurement habits, common/bad, 80

asking people to simply sign of, 83

brainstorming KPIs, 82–83

comparing this month to last month, 85–86

rushing to data and dashboards, 84

using educating, resourcing or funding as improvement initiatives, 86–87

using measures to judge, 80–81

using reports to CYA (cover your arse), 84–85

using weasel words, 81–82

what to replace them with, 87–88

performance measurement library, adding to your, 339

performance measurement methodologies, 28–32. See also PuMP Performance Measure Blueprint

performance measurement process, 18, 32–33

describing your, 32

symptoms of a broken, 42–45

Performance Measurement Process, 23. See also PuMP Performance Measure Blueprint

performance measurement projects starting small and focused, 56–60 See also projects

performance measure(s), 98

better ways to find, 158–60

creating them oneself, 70–71

defined, 15–16

essential features/qualities, 12–15

evaluating potential, 166–67, 168t

vs. events/milestones, 7–8

a few vague words don’t make a, 10–12

implementing, 25, 67, 91–92, 197–240 (see also PuMP implementation)

vs. initiatives, 5–6

listing potential, 164–66

meaningfulness (see meaningful measure design; meaningless measures)

vs. measures of activity completion, 8–9

most measures never improve performance, 304–15

nature of, 4–5, 16–20

reasons for wanting, 341

scope of, 217–18

vs. sources of data, 9–10

terminology problem, 4, 18–21

traditional approaches to selecting, 146–58

adopting measures from industry benchmarking, 151–54

brainstorming, 146–49

vs. Designing Measures using PuMP, 172–73

looking to existence data or legacy measures, 149–51

measuring what stakeholders want to measure, 154–56

outsourcing the selection of measures to experts or consultants, 156–58

reasons for these approaches fail, 146–58

See also KPIs; specific topics

performance owner, 229–31

responsibilities, 230–31

Performance Prism, 29–31

performance reports and dashboards bad habits related to, 84–85

designing good, 249–51, 271–72

(see also PuMP Designing Performance Reports technique)

formatting should make interpretation fast and easy, 269–70

layout should make navigation fast and easy, 268

problems with, 242

ad hoc and hard to navigate, 243

cluttered and cumbersome with too much detail, 243–44

messy and unprofessional, 247–48

tables and graphs cause misinterpretation, 244–47

time-consuming to produce, 249

usefulness and usability, 92

performance result

understanding the process producing a, 322–23

See also goals

performance review forum, redesigning your, 335

performance targets. See reaching performance targets; signals; targets

Pink, Dan, 81

planning process, 17

process benchmarking, 151–52

productivity, focus on, 60

program logic method, 31–32

program management, 106

project management vs. performance management, 106, 331

projects, 99. See also performance measurement projects

proximity (visual design principle), 248

PuMP approach

social elements, 24

technical elements, 23

PuMP Blueprint. See PuMP Performance Measure Blueprint

PuMP Blueprint Pilot Project, eight-week. See PuMP Pilots

PuMP certification, 342–43

PuMP Consultants, 70

PuMP Designing Measures technique compared with traditional approaches of selecting measures, 172–73

advantages, 172–73

disadvantages, 173

principles underlying, 159–60

Step 3.1: Begin with the end in mind, 160–61

Step 3.2: Be sensory-specific, 161–64, 165t

Step 3.3: Find potential measures, 164–67

Step 3.4: Check the bigger picture, 167–70

Step 3.5: Name the measure(s), 170

See also community participation measures; PuMP Measure Design template

PuMP Designing Performance Reports technique, 250–52

Step 6.1: Structure to strategy, 251–52

Step 6.2: Answer what, why, and now what?, 252–54, 255f

Step 6.3: Use graphs that signal, 254, 256–67

Step 6.4: Design to engage, 267–70

Step 6.5: Automate, 270–71

See also performance reports and dashboards: designing good

PuMP implementation

checklist to plan your, 340–42

importance of maintaining momentum through, 55

leaders supporting, 76

resources for success, 337–41

time commitment from the people involved in, 76–77

See also performance measure(s): implementing

PuMP Interpreting Measures. See Interpreting Measures

PuMP Measure Design template, 24–25. See also PuMP Designing Measures technique

PuMP Measure Gallery technique. See Measure Galleries

PuMP meetings, 76

scheduling of, 64

Week 1: Get prepared, and warm up team with PuMP Diagnostic, 65

Week 2: Make the organisation’s strategy measurable, 65–66

Week 3: Design (don’t brainstorm) measures, 66

Week 4: Engage stakeholders, and build buy-in, 66–67

Week 5: Define the ‘bring to life’ details of measures, 67

Week 6: Design useful and usable performance reports and dashboards, 67

Week 7: Respond to what the measures are really telling us, 68

Week 8: Use performance measures to improve performance, 68

PuMP Mind Map exercise, 46

Step 1: Explain the purpose of meeting, 46–47

Step 2: Discuss and collate problems with performance measurement, 47–50

Step 3: Debrief the discussion, 48, 51

PuMP Performance Measure Blueprint (PuMP Blueprint), 20, 22–28, 25f, 33

applicability to particular industries/sectors/cultures/businesses, 20, 22–28, 25f, 33

expert facilitators, 70

vs. other performance measurement methodologies, 28–32

steps/activities in, 23–27, 64–68 (see also specific steps)

See also see PuMP Pilots

PuMP Performance Measure Blueprint Online Program (PMBO), 344

PuMP Performance Measure Blueprint Workshop (PMBW), 343

PuMP Pilots, 54

choosing the focus for the first pilot, 58–59

tips for, 59–60

when organisation does not have defined strategic direction, 59–60

when organisation has defined strategic direction, 59

eight-week length, 54–56, 69

forming the Measures Team, 60–64

how many to focus on simultaneously, 71–72

minimum conditions for success, 75–77

nature of, 56–58

planning, 77

questions frequently asked at outset, 68–77

scheduling (see PuMP meetings: scheduling of)

signs that one should quit before starting, 74–75

writing them up as case studies, 58

PuMP Professional Program, 344

PuMP Professionals, 70

PuMP Reaching Performance Targets framework. See Reaching Performance Targets framework

PuMP Report Design technique, 26

PuMP Results Map. See Results Map

pumpblueprintbook.com, 338

R² statistic (coefficient of determination), 263, 277, 280–82

reaching performance targets, 26, 68, 303, 335–36

obstacles to, 26, 304

choosing initiatives and targets before performance measures, 305–6

judging rather than learning, 311–13

knee-jerk reacting to data, 307–9

mistaking excuses for reasons, 309–11

no routine forum for using measures to manage performance, 306–7

treating symptoms rather than causes, 313–15

See also under signals

Reaching Performance Targets framework, steps of, 316

Step 8.1: Set sensible targets, 317–20

Step 8.2: Prioritise the performance gaps, 320–22

Step 8.3: Find the causes, 322–28

Step 8.4: Choose high-leverage solutions, 328–31

Step 8.5: Look for signals and check for impact, 331–34

readability of data, 202

relevance of data, 201

reliability

of data, 201–2

of performance measures, 13–14

reliability measures, 40

repetition (visual design principle), 248

Report Design technique, 26

reporting of measures, 25–26, 67, 241–72. See also under performance measurement: common struggles with

reports, performance. See performance reports and dashboards

representative data, 201

research outcomes, 11–12

resource-effectiveness of data, 202

results, teasing out the, 120, 121–22t, 123

Results Map, 63, 137, 142

anatomy, 125

bubbles, 127

layers, 125, 126f, 127

lines, 127–29

compared with Balanced Scorecard Strategy Map, 129–30

creating, 124–25, 135

examples, 126f, 128f, 143f, 190f

testing the logic of, 130–35

Retention Rate, 148

revenue, 300–301, 309

Revenue Earned per Employee performance measure, 243

Revenue performance measure, 253, 300

Roosevelt, Theodore, 99, 100

Rothfuss, Patrick, 95

routine variation, 284

SAIDI (System Average Interruption Duration Index), 205–7, 206f, 207f

scatter plot, 259t

Schroth, Richard J., 84

scope of measures, 217–18

self-assessment, questions for a quick, 27–28

Selk, Jason, 303, 310, 329

Sewall, Dave, 99–100

Shewhart, Walter A., 285, 286

Shewhart individuals control chart. See XmR charts

short run (XmR charts), 298, 299t

signals (from performance measures), 320

can be seen only when we filter out the noise, 284–86

defining the signals to respond to, 319–20

to look for that tell us performance has changed, 298–99, 299t

Signal 1: Performance has reached or exceeded target, 224–25

Signal 2: Performance is improving quickly enough to meet target, 225–26

Signal 3: Performance is not improving quickly enough to meet target, 226–27

Signal 4: Performance is stable and not changing, 227

Signal 5: Performance is getting worse, 227–28

Signal 6: Performance is unpredictable or chaotic, 228–29

why we need at least 7 points to qualify as a signal, 300–301

See also interpreting signals from performance measures

‘soft skills,’ 187

solutions

keeping them focused on root causes, 330

relentless focus on, 303, 310, 329–30

special causes. See outliers

spin-off benefits, 163

Spitzer, Dean R., 3, 83, 158

staff engagement

conceptions of, 148

measures for, 147, 148

ways of facilitating, 177–78

adding instead of taking away choices, 182–83

avoiding trampling their creative energy, 181

collaborating to improve measures instead of judging them, 180

helping them articulate their goals instead of dictating them, 179

holding space for their discovery instead of educating them, 178

letting the team fix the causes of results instead of blaming them, 182

letting the team implement their own, 180–81

teaching them to fish instead of giving them measures, 179–80

trusting team’s numeracy skills instead of dumbing down the measures, 181

See also Measures Team: engaging

Staff Turnover performance measure, 148, 208–9

stakeholders

engagement of, 66–67

measuring what they want measured, 154–56

statistical thinking, using statistical techniques without, 284–86

strategic direction, corporate

alignment of teams and business units and processes with, 131–32

how to achieve, 133–34

strategic plan, 305

improved layout of, 306, 307t

typical layout of, 305, 306t

strategic theme, 97

strategy(ies), 74, 98–99

alignment of goals with the rest of the, 119–20

assessing the measurability of, 90, 96–99

expressed as a series of actions, 105–6

how to make it meaningful, 115–16

lack of rigour and logic, 110–15

revealing flaws in the logic underlying a, 133

steps to make it measurable, 115–16

Step 2.1: Test the strategy’s measurability, 116–20

Step 2.2: Tease out the implied performance results, 120, 121–22t, 123

Step 2.3: Map the relationships among results, 123–30

Step 2.4: Test the logic of Results Map, 130–34

Step 2.5: Highlight priority results worth measuring, 134–35

why most strategy is immeasurable, 96–99

System Average Interruption Duration Index (SAIDI), 205–7, 206f, 207f

Talk Time performance measure (call centres), 41. See also Average Handling Time (AHT) performance measure

targets

target setting requires understanding why performance is falling short, 328

See also reaching performance targets; signals

team engagement. See Measures Team: engaging; staff engagement

teasing out the results, 120, 121–22t, 123

templates, 338

time-series graph, 289

total quality management (TQM), 316

Training Failure Processing Time, 318–19, 319f, 321, 324, 325f, 326f, 331–32, 332f

Training Failures process

‘as-is,’ 322–23, 323f

new, 329, 329f

Transforming Performance Measurement: Rethinking the Way We Measure and Drive Organizational Success (Spitzer), 3

trend lines, 45, 163, 276–78, 277f, 278f, 280–81

trend-over-time comparisons, 221, 222, 223t, 260, 260f, 261f, 276–78, 288

Tufte, Edward, 268

Turnover, 148, 208–9

vague ideals, 102–4. See also weasel words

validity of data, 200–202

variability of measures, 38–39

variation, routine vs. abnormal, 284

vision statement, 97

visual design

benefits of professional, 248

principles of, 248

volunteers, working with, 57

waste in performance measurement process

methods of avoiding, 209–11 (see also specific methods)

sources of greatest, 199–200

arguments about data or measure validity, 200–202

measures don’t drive performance improvement, 208–9

measures that misinform and mislead, 202–4

no one is accountable or responsible, 208

reacting to invalid signals and missing important signals in the data, 205–7

too many conflicting versions of the truth, 204–5

symptoms of, 199–200

Watson, Don, 10, 81, 100

weasel words, 10–11, 99–104

lists of, 81, 100–101

and measurability of goals and strategy, 117–18

multi-focus goals and, 106–7

Wells, Herbert G., 285n

Wheeler, Donald J., 85, 273, 290, 293

Whitehead, Lorne A., 175

Wilks, Samuel, 285

Womack, James P., 86

XmR charts, 223t, 286, 301, 318

advantages, 39

best way to display performance measures, 289–93, 293f, 294f

anatomy, 290–92

creating, 293–94, 301–2

description and overview, 206, 259t, 285–86

examples, 291f, 296f

signals to look for in, 295, 297, 298, 299f, 334

Stacey Barr is an internationally recognised specialist in organisational performance measurement and creator of the PuMP® methodology. She has directed her own company since 1999, helping organisations of all shapes and sizes around the world to meaningfully measure what matters.

Stacey challenges the long-held beliefs and bad habits that cause the common struggles people have in choosing, reporting, and using performance measures to improve organisational performance. She helps people get tangibly clear about the results they really intend to achieve in their business or organisation and to recognise how well they are achieving those results.

Performance measurement is the best tool to do this. Done meaningfully and engagingly, it is the most powerful way to focus and motivate people to achieve organisational performance excellence.

Stacey balances her work schedule with her passions for running, exploring nature, and indulging in dark chocolate. She lives in a semi-rural Australian village where wallabies graze on the front lawn and goannas and echidna roam the back yard.

Contact Stacey by email at info@staceybarr.com or visit her online at www.staceybarr.com.