Share:

It’s tempting to have software automate the signals in our KPI XmR charts, but before you do, be aware of these risks and recommendations.

XmR charts are my favourite chart for tracking performance measures. They give us a big enough picture to understand true changes in performance over time, and they give us valid signal rules to know if, when and by how much performance truly does change.

This article assumes you already know what XmR charts are, and how they work. But if XmR charts are a new concept to you, I recommend you read these articles first (they will be a revelation!):

• Three Things You Need On Every KPI Graph
• How Many KPI Data Points Do You Need To Use And Interpret Them?
• How to Build an XmR Chart for Your KPI

A question I am asked a lot is whether the XmR chart signals can be automated. The short answer to this question is yes, they can. But a more important question to explore, before we go ahead and do it, is whether we should automate those signals.

My personal decision is that I won’t automate the signals. And XmR chart guru and author of “Understanding Variation: The Key to Managing Chaos”, Dr. Donald Wheeler, tells me he still takes the same position:

Process behavior charts are the interface between our data and our brains. Nothing is going to happen until our brains are engaged. Software can create the charts, but we should never depend upon the software to interpret the charts. The charts have to be interpreted in context, and the software never knows the context. — Dr Donald Wheeler

If you’re tempted to have software interpret your performance measures on your behalf, please read this article before you do!

How can XmR chart signals (and recalculations) be automated?

One reason why XmR charts can transform conversations about performance is that they make interpretation of our measures happen faster, with more accuracy and less disagreement. That’s because their signal rules are based in mathematical statistics and not opinion.

Because the signal rules are based in mathematical statistics, there is logic behind them. And where there is logic, there can usually be coding. Thusly, it’s a piece of cake for any programmer to set up the code that will look over a performance measure’s XmR chart, and detect those common signals.

The same goes for the recalculation of the Central Line and Natural Process Limits, in the case of a signal of change, because they too are each based on a specific formula.

Michel Dekker’s team, of Nova Silva in the Netherlands, has done just that, in the Microsoft Power BI XmR chart tool they have developed. Watch a short demonstration here:

The advantages of automated signals are easy to appreciate.

What are the advantages of automating XmR signals?

We generally automate things that are repetitive and rule-based, to free up our mental space for harder thinking tasks. That makes it only natural to see this same opportunity for XmR charts. When it comes to interpreting our performance measures, automating the signal rules can:

• save us a minute or so of searching for the signals manually
• alleviate the need for all users of the measures to learn the XmR chart theory
• reduce our mistakes in applying incorrect rules, or applying rules incorrectly, to our measures
• speed up the display of signals and associated traffic lights in our performance dashboards (since manual intervention isn’t require)

But to make the decision about whether to go with signal automation or not, we need to contrast these advantages with the risks of having software interpret our measures for us.

What are the risks of automating XmR signals?

The main reason for my hesitation, and Don Wheeler’s hesitation, to automate XmR signals is that software can never understand context. In fact, Don sums this up perfectly, as his first rule principle for understanding data:

“Data have no meaning apart from their context.” — Donald Wheeler, Understanding Variation [page 13]

Context matters in XmR chart interpretation. Here are several reasons why:

• Due to random variation, a signal in an XmR chart might look like it starts before the cause of the signal actually happened in reality.
• Some improvement initiatives take time to bed in, which we see as a gradual trend in the XmR chart. Automation will produce a series of short signals over the trend period, but the meaningful signal of change doesn’t really happen until after the bedding-in trend levels off.
• If a performance measure is in chaos, there isn’t enough stability to apply XmR chart rules. But software has no way to detect chaos.

These reasons culminate into a significant risk of automating XmR chart signals: the software fails to match signals to real-world explanations.

I don’t expect you to adopt my approach to never automate XmR chart signal rules. But I do invite you to do is be careful how you use them.

How should we use automated XmR chart signals?

It takes a human eye plus knowledge of the system being measured to realise when an automated signal isn’t appropriate. If you do choose to use automated signals, this procedure will help to mitigate the risks and ensure you accurately interpret what performance is really doing:

1. Decide if your performance measure is stable enough for an XmR chart, and if it’s not, focus on standardisation of the process being measured, until your measure shows stability.
2. Look at the automated signal in your XmR chart, before any recalculations of the Central Line and Natural Process Limits are performed.
3. Check if the signal makes sense, based on your knowledge of the real-world changes associated with that signal.
4. If the automated signal isn’t starting at the right point in time to match the real-word change, correct it manually.
5. Only after you’re happy that the signal of performance reflects the reality of performance, recalculate the Central Line and Natural Process Limits.

The only right thing for software to do with XmR charts is to give us the flexibility to adjust the signals and set when the recalculations should happen, if the automation isn’t appropriate. Michel Dekker and his Nova Silva team know this, and that’s why their Power BI XmR chart tool gives you complete control over signal detection and recalculation.

All new technology that replaces the more routine aspects of human processing can only do what we program it to do. And so far, BI software has no way of knowing the context upon which accurate interpretation of our data depends. And accurate interpretation is what good decision making requires.

Software automation of KPI signals is merely a starting point that requires our knowledge of context to draw correct conclusions.
[tweet this]