Training Metrics that Matter

In my spare time, I am a cookie artist. I spend many hours combining flavors in my quest to create the ultimate flavors. After each attempt at cookie awesomeness, I evaluate the changes I made and their ability to transform a basic chocolate chip cookie into the ultimate chocolate chip cookie experience. I begin by assessing the reaction from my loved ones and customers. Next, I reflect on what I learned during the process and how this new knowledge will help me in my quest to dominate the cookie world. I determine if the new knowledge is useful and explore how I will apply it. Finally, I evaluate my cookie sales and review the results of the new recipe.

Kirkpatrick Model

This four-step process may sound familiar, and it should, because it is an example of the Kirkpatrick evaluation model. This model has been in use for well over 60 years in the learning and development field. Donald Kirkpatrick was successful in creating a model that convinced organizations learning is intended to enable behavior change and result in organizational excellence. This work was instrumental in getting learning and development a seat at the table in the executive suite.

This four-step process works well for making cookies, but it does not translate well when trainers are required to create learning experiences that result in agents who are capable of making good decisions and have the ability to take the correct action based on their choices.

L-TEM:  Measuring What Matters

Dr. Will Thalheimer has developed an eight level evaluation model, The Learning Transfer Evaluation Model: Sending Messages to Enable Learning Effectiveness (L-TEM), explicitly designed to spur learning professionals to greater learning effectiveness and push them to use learning designs that support long-term remembering and engage in a cycle of continuous improvement.

Levels 1 & 2

Levels 1 and 2 measure attendance and level of engagement during activities. Attending the learning event and engaging in activities are necessary to reach the desired goal; however, they are not adequate measures of learning effectiveness. How many times have you had to wake up a sleepy agent during training? The L-TEM evaluation tool reminds us that measuring attendance and activity completion are not adequate measures to evaluate learning success.

Level 3: Learner Perceptions

Historically, smiley sheets have asked agents to measure their satisfaction with the content, trainer, and the learning environment. Measuring general satisfaction with the material, trainer, and environment often fails to provide actionable insight, so these metrics are not an adequate measure of learning success.

I am not suggesting we stop using the smiley sheet, but I am suggesting trainers choose to develop a better smiley sheet. A Performance-Based Smiley Sheet measures learner comprehension, motivation to apply learning, confidence with the new knowledge, and more. These measures will get us closer to gauging learning effectiveness, but an accurate measure of learning effectiveness requires the trainer to complete all eight levels of the model.

Level 4: Knowledge Checks

Level 2 Kirkpatrick measures the degree to which participants acquire the intended knowledge, skills, attitude, confidence and commitment based on their participation in the training. Too often trainers interpret learning as knowledge recitation and create knowledge checks that ask learners to regurgitate facts and terminology. The modern contact center agent engages in critical thinking that requires competence in decision making, the ability to perform relevant, realistic actions, and transfer of knowledge. Even a well-written test will not measure the agent's ability to perform on the contact center floor.

Level 5: Decision Making Competence

Dr. Thalheimer reminds trainers that at a minimum, learning experiences should be designed to support realistic decision making and we should evaluate our learning experiences based on how well they support learning in gaining decision making competence. Measuring decision making capability begins during training (short-term remembering) by presenting them with realistic situations and requiring them to make decisions that are similar to the types of decisions they will have to make on the contact center floor. How? Begin with the end in mind. Ask what decisions agents will be required to make and build scenarios that replicate those customer service interactions. If you are new to building scenarios, visit Cathy Moore's website for resources, blog posts, and courses on scenario building.

How many times has an agent demonstrated excellent decision making competence during new hire training and then failed a significant number of QAs? Dr. Thalheimer reminds us that "Measuring short-term realistic decision making is a giant leap from earlier levels, but it is still insufficient as a learning metric because it does not evaluate whether learners can maintain their newly-learned decision making competence over time-to support them in using their skills on the job." Long-term remembering must be measured and supported. During the design phase of a learning experience, begin from the bottom of the model and move up. Ensure that your QA process measures decision-making competence and build learning interventions when decision making competence is not being met.

Level 6: Task Competence

Task competence comprises decision making, as discussed in level 5 of L-TEM, but it also adds task performance. Suppose we teach agents to provide tech support to customers. We assess their decision making competence and ask them to demonstrate their questioning skills. Finally, we ask them to search the knowledge base quickly and efficiently. If we do that, we're measuring their decision making competence. What if we take it a step further and assess their ability to match the tone of the customer, demonstrate empathy, and take ownership of the problem? If we do that, we measure their ability to take action after they've made a decision. Dr. Thalheimer reports that in measuring both decision making and action taking, we measure task competence. Level 6 provides practice in all the real world behaviors needed for agents to be fully effective in their work.

Level 7: Transfer

Measuring for learning transfer requires trainers to select a relevant performance situation to target. Again, begin with the end in mind. Decide when agents will be required to use the new knowledge, and measure transfer accordingly.

Level 8: Effects of Transfer

We train agents not just so they transfer their skills to their work, but because we expect them to create effortless experiences for customers. That is the business objective, but what if we create learning experiences that benefit the learner in all areas of their life? This strategy will increase the engagement of the agent when they understand how this new knowledge will impact all aspects of their life.

In Dr. Thalheimers' Learning Landscape Model he discusses the importance of acknowledging the outcomes that affect the learner. Identifying how learning transfer positively impacts coworkers/family/friends, the organization, the community, or society will encourage the agent to engage in the learning process and transfer the new knowledge in many areas of their lives thus increasing the transfer on the job.

Continuous Improvement

A well designed QA process will guarantee a constant improvement process if it is augmented with consistent communication between training and quality.

Set your agents up for success and measure metrics that matter. Review your current QA process and ask these questions:

• How often are quality and training meeting to discuss the needs of the organization?
• Does your QA process measure the quality of the agent's decision?
• Does your QA process measure decision making and action taking?
• Have your QA professionals received training on how to light up the brain for increased learning?

Together, quality and training have the power to create learning experiences that will result in engaged agents who are capable of making decisions and taking actions that will result in effortless experiences for customers.

Metrics Matter

It is time to measure what matters; decision and task competence, long-term remembering, transfer, and effects of transfer. Kirkpatrick works well if you are on a quest for the perfect cookie but does not measure what matters if your agents are required to make complex decisions and take action on those decisions.

Begin with the end in mind. Design your next learning experience with Dr. Thalheimer's Learning Transfer Evaluation Model by your side. Use his evaluation tool to measure the metrics that matter and revolutionize training in your contact center.