Share:

The PuMP Results Map facilitated a paradigm shift in the approach recommended by the UN Economic Commission for Europe to measure the value of official statistics.

My career started as a research statistician. I contributed to the collection, analysis and reporting of official statistics on topics ranging from broadacre agriculture to car racing. The value of statistics was clear to me. And it still is. But I still find, to this day, that many users of official statistics are not as convinced of their value.

In their October 2022 report Measuring the Value of Official Statistics: testing and developing a measurement framework, the Conference of European Statisticians Task Force offered a very sound theory as to why: ‘value’ is too abstract.

In fact, their entire report is a brilliant case study of why this is a fundamental obstacle to meaningfully measuring the value of official statistics (and measuring the value of anything). Even though the Task Force began with a brainstormed list of possible indicators to measure the value of official statistics, that’s not where they ended up.

“The Task Force started with an indicator set before realizing this was precisely the wrong place to start. Instead, it shifted to a focus on clarifying the concept of value, leading to a broad, rather than narrow, conception of value.” – Ed Humpherson, Director General for the United Kingdom’s Office for Statistics Regulation

Central to their clarification of the concept of value was PuMP’s Results Map. That’s exactly what a Results Map does: it clarifies the results that are most important to measure and improve, unpacking them in a relationship map that unifies them in telling a complete story of a strategy. (You might want to read more about a Results Map before you continue, if you’re not already familiar with them.)

“A paradigm shift is needed to develop measures that work outwards from a central goal. A Results Map defines a clear path to central goals of official statistics, that works outwards through organizational strategies helping to achieve goals, tactical outcomes and measuring areas for which indicators can be developed.” – The Task Force

Some of the observations the Task Force had with their experience with the Results Map are very typical of any of our PuMP Alumni:

• A Results Map should be created by the right people: senior management and strategy professionals to build the inner layers, and technical experts (and for most situations, the people working in the processes) to build the outer layers.
• Building a Results Map is “intensely challenging, but enlightening”. It’s paradigm shift, and so the first few attempts at it will include a battle with old habits.
• A Results Map can be like a filter, to remove measures that really are not relevant to the goals that matter. The Task Force noticed how uncomfortable it can be to let measures go.
• Results Maps can look messy because the relationships are not unidirectional or linear. Rather, many results can relate to any other single result.

And their conclusions are definitely easy for me to endorse also:

• The Results Map is more than just giving visual clarity to a set of goals or strategy. It is a filter to make sure only measures that are meaningful to that strategy are included.
• Communicating this clarity and focus on meaningful measures is another important capability of the Results Map.
• Building a measurement framework with a Results Map means that each organisation can refine and tailor their own goals and measures over time, without starting from scratch each planning cycle.
• Measuring value – and any results – in this way helps with both uses of measurement: to prove what’s working (for external communication) and to improve what needs to work better (for internal strategy execution).

Measuring the Value of Official Statistics: testing and developing a measurement framework is a report not just about measuring the value of statistics. It describes a process for how to measure the value of anything, by shifting the paradigm to measuring results. And it shows the dysfunction of starting with brainstormed measures, and how the clarification of our goals must come first.

A special thanks to our PuMP alumnus, Giles Sullivan of New Zealand, who was instrumental in introducing the Results Map to the Task Force.