Share:

Jack Phillips’ 6 levels of measures for training is an excellent framework. Here is an example of how to use that framework to measure the effectiveness of training, beyond just satisfaction.

That’s where training evaluation began, with ‘happy sheets’. The questionnaires that participants in a training event filled out at the end of the course, usually when they were exhausted, jammed with new knowledge and keen to pack up their binders and escape.

What is a happy sheet really measuring anyway (even if it’s a good one)? Irrespective of what questions you ask –

• was the trainer easy to understand?
• was the content relevant?
• was the environment conducive to learning?
• do you intend to apply what you learned?

– you’re just measuring the participants’ perspectives. And that won’t give you any idea of what kind of impact it will – or will not – have on your business, after the freshly trained people return to work.

What are the results that matter in relation to training?

A nice and simple model comes from Jack Phillips, author of How to Measure Training Results : A Practical Guide to Tracking the Six Key Indicators, where he suggests we look at an evolution of a model originally proposed by Donald Kirkpatrick. There are six levels to this model:

1. reaction and satisfaction
2. learning
3. behaviour
4. results
5. return on investment
6. intangible benefits

LEVEL 1: Reaction and satisfaction

Reaction is the satisfaction or perceptions of the trainees, and therefore is more often an evaluation of the trainer and training process and not so much an evaluation of the impact of the training. This is where the happy sheet fits in: but a well designed one!

For example, when I run my PuMP Performance Measure Blueprint Workshop, I am very interested in how my customers react to their training experience. I don’t use happy sheets, but ask four simple questions:

• Q1 Overall, how do you rate your experience of this workshop?
• Q2 Why didn’t you rate any higher?
• Q3 Why didn’t you rate any lower?
• Q4 How likely will you refer this workshop to others?

This information helps me understand what to keep in the workshop, what to change, and how well it’s performing as time goes by.

LEVEL 2: Learning

Start with a clear definition of the learning objectives for the training, and what the evidence is as to whether the objective is met or not, or the degree to which it is met.

For example, an objective of my PuMP Blueprint Workshop is being able to interpret an XmR chart. The evidence of how well someone achieves this objective is found by checking their recognition of the correct rules for interpretation in contrast to incorrect rules. We gather this data through the PuMP Certification Exam.

LEVEL 3: Behaviour

The third level in the training evaluation model is behaviour, or implementation or application of the learning.

This is tricky from the perspective of the trainer, since after the learners leave the training situation, they are rarely in contact with the trainer again.

To some extent, I try to assess the implementation of my PuMP Blueprint Workshop customers through research of a random sample of participants, and through the PuMP Pilot implementation dashboard in the PuMP Community Website.

For a manager or supervisor, it’s easier to find out if the learning is being applied back at work. But it does require you set up an implementation plan of some kind and monitor it regularly to not only check if behaviours are changing as a result of the training, but also to keep it top of mind, to facilitate the new behaviours to become new habits.

LEVEL 4: Business Impact

The fourth level of training effectiveness is the business impact. And this is where your business performance measures take on an important role. As you know, from a manager’s point of view, training is a business improvement initiative. It’s a change they are trying to make to improve some aspect of business performance.

They might send their staff to negotiation skills training to improve the sales close rate. Or send their staff to safety training to reduce injuries and lost time at work.

When you’re clear about the reasons for the training, you can do one of two things. Firstly, you could choose the few performance measures you currently have that monitor the performance areas the training should improve. For example, if you wanted to improve staff’s negotiation skills to improve the sales close rate, you’d use your measure of % of Quotes Accepted or your equivalent. Or if you were training to improve safety, you could use your measure of Workplace Injuries, or Days Lost Due to Injuries.

But if you aren’t currently measuring the area of business performance you think the training will improve, then you really have to start measuring it!

LEVEL 5: Training ROI

The fifth level in the training evaluation model is return on investment, or ROI, of the training.

In it’s basic form, the Training ROI measure has a numerator and a denominator. The numerator on the top of the ratio is the “return” part of the return on investment. It’s usually calculated as the financial benefit minus the costs associated with the training. The denominator on the bottom of the ratio is the “investment” part of the return on investment. It’s the costs associated with the training.

Training ROI is an advanced measure because you need to have a very good handle on the financial benefit of your training, and it can take some time for this to fully manifest.

If you’ve done your homework, you will have already scoped the intended financial benefits of training at the stage where you created the training objectives, or made the choice to conduct that particular training.

And you’ll have to deal with situations where the training objectives are more intangible, like reducing stress or improving communication skills. In these cases, you need to extrapolate beyond the training objectives, asking questions like “where would costs be saved or revenues increased if communication improved?” Some estimating might be in order, but it needs to have some rigour to it, so don’t do it just for the sake of an ROI measure.

ROI also presupposes you have a good handle of the costs of training. And this is not just the course fee – it’s the lost opportunity from staff not being at work, their wages, travel costs and the time to research and prepare for the training in the first place.

LEVEL 6: Intangible benefits

The sixth and final level in the training evaluation model is called intangible benefits, but I don’t think this level has been named appropriately.

Basically, any result you can describe in sensory-rich language becomes observable, and therefore tangible. When it’s observable it’s able to be measured in some quantity.

When you read Jack Phillips’ book, you can see he’s actually describing what I call spin-off benefits. These benefits are seen in the run-on effects of the training being a success. They aren’t directly targeted by the training.

For example, you might train people in the Lean methodology to directly target lower costs and shorter cycle times. But a spin-off benefit could be that safety improves, or employee morale increases, due to greater control over work quality.

Using the 6 levels of training measures

When you’re going to review how well your training process is working, you ought to use all levels of the training measures together. What story do they collectively tell of how training is working?

And you’ll want to consider what other measures of business performance to review at the same time, too. Particularly if other results have had a negative impact that has diluted or even sabotaged the training results.

I’d also like to highlight that we haven’t talked about training efficiency here. Things like measuring the number of people successfully trained per training event or per training dollar spent. Nor have we measured the relevance of the training – as in, did it have an impact in an important area of business performance? These may be areas you want to explore also, if you’re going to build or renovate your suite of training measures.

TAKE ACTION:

How do you measure training? How do you know if your training budget is adding to the financial, social and environmental bottom lines of your business, or actually eroding them?