SHIFT's eLearning Blog

Our blog provides the best practices, tips, and inspiration for corporate training, instructional design, eLearning and mLearning.

To visit the Spanish blog, click here
    All Posts

    Do You Know How Successful Your eLearning Program Really Is?

    Recently, I attended a Congress of Human Resources. On stage, the speaker appeared charismatic, professional, and he gave quite an entertaining presentation. He held the audience's attention by constantly making jokes and adding interesting personal anecdotes.

    I sat through the entire speech waiting for the speaker to reach a climax, make a solid point or establish a noteworthy conclusion. In the end, I was left unsatiated, with the feeling that while the attendees had all had a good time, no real learning had actually taken place. 

    This same situation occurs during trainings for countless companies. In-person trainings involve a fun and enjoyable classroom environment, but often lack applicable content. Similarly, online trainings are all too often attractive eLearning courses with impressive graphics and animations that still fail to teach meaningful information. 

    We must remember that the ultimate goal of a training course is to learn. But how do we know that we have fulfilled this goal after delivering it? Here, it is important to mention one key element: evaluation. 

    There are many different approaches to evaluating the effectiveness of an eLearning course, but they all share a common first step: identifying success metrics.  Kirkpatrick's taxonomy is one seasoned model that continues to receive widespread use. Developed by Dr. Don Kirkpatrick in the 1950s, the model originally contained four levels of training evaluation. Now, the levels have been clarified by Don, Jim, and Wendy Kirkpatrick to form what is called "The New World Kirkpatrick Model". Since the concept has continued to evolve alongside training, it remains a relevant and robust evaluation framework.

    This evaluation model is applicable to both classroom training and eLearning.



    So to make sure you know how successful your eLearning program really is, check these metrics: 

    1. Metric #1: Reaction to learning

    What it measures: Level of satisfaction, interest and engagement of students.

    This level assesses what learners think and feel about the training or eLearning course. But beyond simple satisfaction, New World Model additions measure Engagement (the degree to which participants are actively involved in and contribute to the learning experience) and Relevance (the degree to which participants will have the opportunity to use or apply what they learned in training on the job).

    The evaluation of this level measures how participants react to a given training activity. Appropriate questions include: "What was your level of satisfaction after participating in this course?", "Did you feel engaged?", and "Was it relevant?"

    How it is done: This evaluation is carried out through reaction surveys, feedback forms or questionnaires, which generally determine the categories to measure and use scales of satisfaction, such as: excellent, very good, good, fair, poor.

    However, these evaluation tools tend to be poorly designed and unnecessarily lengthy. It's also easy to ask the wrong questions, leading to data that might not be useful.  You can read this article to become acquainted with what to ask and what not to ask.

    What information it conveys: This evaluation provides important information about the quality of the course, allowing for immediate adjustments. This could include information about the instructor’s performance, the quality of the materials and the content, etc.

    Who the results are for: Training managers.

    Evaluation goal: The goal at this stage is merely an initial endorsement by participants of the training. This evaluation is meant to identify any significant problems, not to determine the ultimate effectiveness of the training.

    2. Metric #2: Knowledge

    What it measures: What knowledge, skills, attitudes, confidence or commitment were acquired by participants?  

    According to the authors of the book, learning cannot occur without change in behavior. Therefore, evaluating the degree of learning achieved by the participants as a result of their participation in a training process is indispensable.

    How it is done: This evaluation is performed before beginning and after finishing the course, in order to compare the increase in knowledge and assess what changes have taken place. Its purpose is to determine if the participants have actually acquired the knowledge and skills that they were meant to during the training.  

    It generally consists of 1) a written test to measure knowledge and attitudes and 2) a performance test to measure skills.  However, evaluators should consider using scenarios, case studies, sample project evaluations, etc, rather than just test questions.

    What information it conveys: From this evaluation we can detect where the training was successful and where it has failed, in order to plan other techniques or methods to improve the quality and quantity of learning in future programs. In addition, by knowing the extent to which learning has occurred, we can expect certain results in the third level of assessment - namely, a change in behavior.

    Who the results are for: Supervisors, line managers.

    Evaluation goal: Confirmation that learning has occurred as a result of the training.

    3. Metric #3: Behavior change

    What it measures: Do participants use their newly acquired skills on the job?

    This level measures how much learners are applying new knowledge back at their jobs. It measures the change in behavior, or how the training affects performance. It’s all about TRANSFER.

    The New World Kirkpatrick model suggests that to achieve transfer of knowledge, learners need incentives. Incentives, or drivers, require processes which reward and reinforce the performance of certain critical behaviors. These systems not only support learners who apply what they've learned to their jobs, but they also encourage a sense of responsibility among those who successfully perform the behaviors. Ultimately, tangible incentives are the force driving work. Learners who receive real rewards for learning and applying new knowledge to better their performance are more likely to put in the amount of effort required to change their behavior.

    How it is done: This step is done through observation, performance benchmarks, interviews and/or surveys to participants, their immediate superiors, peers, subordinates and others who may be involved or interested in the process of behavioral change. Preferably, you should wait for some time after the training event to take this measurement; the author recommends two to three months.

    What information it conveys: This step allows us to measure if the knowledge, skills and attitudes learned were ultimately transferred to the workplace. And if they weren’t, then it allows us to identify the reasons why and make adjustments. For example, it may be that the knowledge was acquired, but the work climate and conditions are not conducive for putting it into practice - perhaps the boss does not see it as immediately useful, or the worker  hasn’t dedicated enough time, etc.

    Who the results are for: Managers.

    Evaluation goal: To determine that learning has affected behavior or on-the-job performance.

    Here is a step-by-step plan for implementing Level 3 in your training initiatives.

    Also, find more info here.

    4. Metric #4: Results and Impact

    What it measures: Did it impact the bottom-line? Has the course helped improve business results? Did the targeted outcomes occur as a result of the training experience and follow-up reinforcement?

    Organizations often talk about ROI (Return on Investment), but New World Kirkpatrick Model considers ROE (Return on Expectations) a much more useful concept. Unlike ROI's narrow focus on money, ROE's broader focus on expectations leads to a long-lasting result. Kirkpatrick asserts that rather than asking limited questions like "Will I get my money back?", it is much better to consider results-oriented questions, like "What should the result look like?" or "What do we want to change in our organization?"

    It is also possible to think of Return on Expectations as the deliverables resulting from a successful training initiative which demonstrate to key business stakeholders how well their expectations have been met.

    How it is done:  The value that training professionals are responsible for delivering is defined by stakeholder expectations. Therefore, this step is completed by interviewing those involved with the intended impacts. An example is requesting information from supervisors and managers on the number of rotations before and after imparting the training program.

    It is imperative that the training professional and the stakeholders enter a negotiation process, so that the training professional can make sure that stakeholder expectations are realistic and achievable with the resources at hand. Thus, you must ask stakeholders questions which clarify their expectations regarding all four evaluation levels, beginning with Level 4. Since stakeholder observations are typically fairly general, you must convert them into observable, measurable success outcomes. This can be done by asking stakeholders what success would look like to them. These outcomes are then used as the Level 4 results and become your collective targets for accomplishing a return on expectations.

    What information it conveys: This measurement provides tangible evidence or indicators that allow us to nurture training needs.

    Who the results are for: Department managers, execs, CEOs.

    Evaluation goal: Confirmation that the training has had the desired results within the organization.


    Final Note:

    It is useful to note that the four levels can be used in reverse during the planning process to eliminate two problems that are common within many different initiatives - focusing on learning objectives rather than strategic goals, and never actually making it to Levels 3 and 4.

    Free eBook: A Quick Survival Guide for Modern elearning Designers


    Do You Know How Successful Your eLearning Program Really Is?

    Related Posts

    3 Ways to Leverage Artificial Intelligence for Rapid eLearning Course Creation

    In the digital age we're living in, eLearning has experienced explosive growth in the corporate realm. More and more, companies are recognizing the benefits of implementing online training programs for their employees, as they offer flexibility, accessibility, and efficiency compared to traditional training methods.

    6 Proven Principles to Create Memorable eLearning Courses

    Picture this: You've invested countless hours in creating a brilliant, engaging eLearning course. It's chock full of relevant information, interesting modules, and interactive quizzes. You've done your part, but there's one question that haunts you as you stare at your masterpiece:"Will my learners remember any of this a month from now?" It's a fair question. After all, the true measure of successful training is not just the initial understanding, but also the longevity of the knowledge imparted. So, how do we ensure that the valuable information you're providing sticks in the minds of your workforce long after they've logged off the learning platform? Understanding how human memory works is pivotal in this quest. But to truly elevate your eLearning strategies, we need to delve into the ways that can help learners not only acquire but also retain new knowledge effectively. This pursuit leads us to six key premises that can transform your eLearning from a fleeting moment of instruction to a memorable learning journey. Join us as we navigate the fascinating world of learning retention, tapping into cognitive science, cutting-edge research, and proven strategies to help your workforce remember what they've learned. For training leaders like you, this blog will be your guidebook, transforming the way you think about eLearning and its potential to empower your workforce in a whole new way.

    How To Avoid Designing Cluttered eLearning Screens

    How many times have you found yourself sitting through a course, feeling bewildered and lost because you couldn't determine what to focus on? And how often have you strained your eyes during a presentation, attempting to decipher the minuscule content squeezed into a tiny space at the bottom of the screen, from your distant seat in the fifth row? Unfortunately, these dreadful experiences are all too familiar. Even the most well-intentioned instructional designers have been guilty of creating cluttered and chaotic eLearning screens, where the intended message gets drowned amidst a chaotic mix of images, icons, text, charts, and graphs. This mishmash of visual elements not only overwhelms the screen but also tests the patience of learners, hindering their ability to grasp meaningful concepts from the course. To alleviate this issue, here are seven effective strategies to help you declutter your eLearning screens: