Share:

Written by Trevor Hobbs, Assistant to the Town Administrator, for a municipality in Tennessee, USA. Edited by Stacey Barr.

Why aren’t existing metrics bringing world leaders to a consensus about whether the COVID-19 pandemic is improving enough, or not?

This is a question I’m sure lots of PuMP users have asked themselves during the last few months. But with all the confusion and disagreement about what data is telling health officials and policy makers world-wide, I wonder if that question doesn’t need revisiting.

The COVID-19 pandemic, though a truly tragic and serious situation for us all, offers an interesting case study for the value and purpose of the PuMP methodology.

Several months ago, when most of the world hadn’t yet been touched by the Coronavirus, health officials and leaders focused mostly on getting prepared; making sure that we had enough face masks, hand sanitizer, and hospital beds to handle the needs of our communities. Communities stocked up on essentials so they could stay home. Businesses scrambled to get their employees set up to work from home. We did all of these things in order to survive the pandemic and hold onto as much normalcy as possible.

Now, months later, face mask orders, social distancing, limited capacity at stores and restaurants, limits on public gatherings, are all part of our new normal. Though initially considered temporary tactics to slow and prevent the spread of the Coronavirus, we are now wondering when these restrictions will no longer be necessary.

These questions offer us a great opportunity to take a strategic approach to public health amid a global pandemic.

How do we know which are the right COVID-19 metrics?

While I’m partial to it, I can’t help but think that PuMP has a lot to offer to those addressing COVID-19 and making decisions about what to do.

PuMP starts with clarifying our strategic plan, not just creating performance measures. We start by asking ourselves what success looks like: How will we know when our result has been achieved? What will we observe (see, feel, taste, smell, hear, experience)?

We need to start by defining our results, before we start discussing what we are going to DO about anything. If we don’t have a clear definition of our results, we can’t achieve them, and we might as well do nothing at all.

What are the results that matter for COVID-19?

We invite you, the reader, to share your thoughts (in the comments below this post, or email Stacey). To prime your thinking, here are some examples:

• The spread of COVID-19 slows and stops
• The global population has immunity to COVID-19
• People who contract COVID-19 fully recover
• Companies, businesses and organisations don’t go out of business due to COVID-19 restrictions
• Everyone who lost employment due to COVID-19 finds employment again
• [Email Stacey with additional results you think matter]

Are COVID-19 outcomes too intangible to measure?

If we accepted the premise that some outcomes are too broad and intangible to measure directly, then we accept the premise that there will be no evidence of such outcomes if and when they happen. If there is no evidence of an outcome when it happens, no-one will notice if it’s happening or not.

So why bother with all the effort to try and create that outcome in the first place?

The question then becomes, how will we know when the effects of the COVID-19 pandemic are over? And along the way, how will we know that our strategies are having the intended impact on the situation?

Which metrics are being used to monitor COVID-19?

There are many metrics being used in an attempt to monitor the direct and indirect effects of COVID-19, and these are only a few:

• Daily New Cases
• Daily New Deaths
• Total Cases per Million Population
• Total Deaths per Million Population
• Total New Tests per Million Population
• Transmission Rate
• Hospital Capacity Utilisation
• Ventilators Used
• Unemployment Rate
• Consumer Price Index
• Retail Sales
• Domestic Violence
• [Email Stacey with additional metrics you know of]

If we know all this, why is the pandemic still wreaking havoc?

Existing COVID metrics aren’t helping.

A very real and sadly comical interview took place a few months ago between Axios correspondent Jonathan Swan and United States President Donald Trump. They discussed data reporting various information for different countries.

At approximately 12:00 minutes in, they begin discussing the COVID-19 reports. Then at approximately 13:50, they begin to discuss whether it is more appropriate to measure the number of deaths by country as a proportion of positive cases, or as a proportion of the country’s population. While those are two different performance measures that support two separate results, it did get me thinking about what results are implied if one measure is used over another, and why has population been chosen as the default basis for measurement.

With PuMP we understand that it is absolutely essential that we determine our intended results first, before we ever attempt to design a performance measure to support it. If we don’t, we may very likely spend most of our time disputing what our data is telling us.

At about 15:50 they start discussing and disagreeing about the changes they interpret from the data. Can someone please use a Smart Chart already?! I think there’s a lot of noise in the reports circulating, but none of the reports are showing whether statistically significant change has occurred.

Is the COVID-19 pandemic improving?

Trend lines are great for certain purposes, but when we want to track change over time, particularly when trying to determine whether our initiatives are having the intended impact on our results, is to use a Smart Chart. Smart Charts allow us to visualize whether central lines have shifted, identify outliers, and see when our normal range of variation has shifted.

Many public health authorities have established benchmarks for the kinds of trends they want to see in order to loosen, retire, or tighten restrictions. But few of them come close to capturing the essence of a performance measure. Without a clear vision for success, data can only tell us so much, and we will struggle to answer this question.

Luckily, PuMP offers us solutions to these problems.

What do we do now?

With great power comes great responsibility.

Those who are blessed with the knowledge of PuMP and the foundational principles of Performance Measurement have the responsibility to share their perspective with those who need it. Leaders and Public Health officials are working hard and doing their best to stop the spread of the Coronavirus, and they need the help of Performance Measurement specialists to construct a framework for their strategic plan. The Coronavirus may be outside of our control, but there is a lot within our circle of influence.

Part II of this article will demonstrate how PuMP can help to solve the problems we face with managing the impacts of COVID-19, by claryifying, measuring and interpreting some of those impacts.

Which COVID-19 metrics are best? We can’t answer that until we know the specific pandemic results we’re trying to improve.
[tweet this]