Evaluating Hybrid & Virtual Learning: Defining the Trend

5 Recommendations to Evaluate Hybrid and Virtual Learning, and What Virtual Learning Experts® Need to Know. 

This is part of an ongoing column by Virtual Learning Expert® Jennifer Finan. She’s exploring trends that impact virtual classroom trainers and designers to improve learner engagement in the hybrid virtual classroom.

Is Evaluating Hybrid and Virtual Learning The Answer For Engaging Learners Online?

Organizations frequently spend months designing, developing, and then delivering training programs intended to improve performance in the workplace. Few organizations can honestly, objectively say exactly what impact their training programs have actually had. By evaluating the efforts taken for learning in a virtual classroom, training teams can receive valuable feedback. This feedback allows for further improvements and adaptations to better suit the virtual learner and increase overall online virtual classroom engagement.

What Do We Mean By Evaluating Hybrid and Virtual Classroom Learning and Why Is It Trending?

Impact Analyst, Scott Weersing defines evaluation as, “The practice to prove and improve the impact of learning programs on business outcomes.” Therefore, we need to prove that the training was responsible for the improved business outcome and that the results cannot be attributed to some other factor. We can only prove that through effective measurement and evaluation. We also need the evaluation information in order to improve the programs and virtual classroom experience. So we need to know, not guess, what works and what doesn’t.

Training evaluation is always a popular topic. With the recent transition of so many organizations moving to a virtual first or hybrid working approach, it’s important to evaluate the efficacy of training in these new environments. In addition, economic pressures mean many practitioners need to prove the value of training to the business. This must be done through evaluation.

How Can We Use What We Know About Learner Engagement To Evaluate Hybrid and Virtual Learning?

When it comes to evaluating learner engagement, Dr. Charles Dye says, “In every distinct learning environment, the methods and means of measuring learner engagement are different”. He outlines why it is important to measure learner engagement in this blog post.

To keep it simple, we can use the dimensions of virtual classroom engagement as a framework to ensure we ask the right questions to develop improvements and strategies for virtual learning. 

Emotional

Whilst this is often seen as just a ‘Level One (Reaction)’ evaluation, it’s important to ask how learners felt about their training experience. Did they enjoy it? Did they feel it was valuable? Did they feel safe enough to ask questions and share their experiences? These are great questions for assessing how well the program works for engaging students in the virtual classroom.

Environmental

These questions are even more important in hybrid working. Consider including questions to help identify if there are any environmental barriers to learning. For example, ask learners where they joined from and if they experienced any issues connecting, hearing, or participating in the virtual learning platform. This will help iron out any technical issues for future programs.

Intellectual

These are critical questions that help us answer whether the virtual classroom training was impactful at all. Consider asking questions that allow learners to demonstrate that they learned something. Ask questions that assess whether the training met the learning objectives and supported the learner to apply that new learning in the workplace. These questions will typically be on Level Two (Learning) and Three (Behavior) in The Kirkpatrick Model. 

Level Four (Results) type questions may be better directed at leaders in the organization who have better visibility of the impact of the training on the business. If your organization wants to go further, The New World Kirkpatrick Model has additional levels and The Learning Transfer Evaluation Model takes it even further. Find out more about various evaluation models here.

5 Recommendations For Evaluating Hybrid and Virtual Learning

1. Ask the right questions. Adam Connor, one of the authors of Discussing Design, suggests there are three types of feedback: Reaction, Direction, and Critique. The first two can be useful when we want to know what the learners felt about the training generally and what they would improve. The third type, Critique is the most helpful when it comes to improving our training programs. We want evaluators to use critical thinking to decide whether the program actually meets the objectives it was intended to meet. To get useful feedback, we need to ask the right questions. Too often, learning professionals ask questions that only relate to reaction or direction. 
   
   Asking if the learner enjoyed the experience and what they would recommend to improve it next time can be useful, but nowhere near as useful as whether the training met the objectives or performance goal. In this podcast, Karen Vieth says, “I don’t really care if my audience liked the training or not, it’s: is it impactful… We care if we are making an impact on learning, and how they’re transferring that learning back on the job.”
   
   
2. Ask at the right time. Typically, organizations ask learners to evaluate the training immediately after they’ve experienced it. Is this the best time for an evaluation? If you’re shopping for a dog toy and you read the reviews, you’ll find lots that say the toy ‘looks’ or ‘seems’ indestructible. This is because the review has been written 5 minutes after the item arrived – that’s when the email comes in asking customers for a review. 
   
   However, the most useful reviews are from customers who have let their dogs play with it for at least a few days! How indestructible is it really? The same applies to training programs. They often feel very ‘useful’ or even ‘impactful’ 5 minutes after class has finished, but how useful or impactful are they once the learner is back in their job? That’s when we want to ask.
   
   
3. Ask the right people. Consider who you want input from. Of course, we want to hear directly from the learners themselves, but perhaps it would be useful to hear from those managing virtual teams. The learners’ managers may be able to see how the training is being applied in the workplace and may be better placed to evaluate the impact the training has had on performance. 
   
   In addition, seek feedback from the Virtual Learning Experts® involved. They have a ‘front row seat’ throughout the program and will be able to spot where challenges are and where learners may need some additional support. The instructional virtual training team will be able to identify opportunities for improvements along the way.
   
   
4. Analyze responses. Before pulling together the evaluation questions, consider how the data will be analyzed. This can be particularly challenging when asking questions that require a ‘free text’ type response. These are not as easy to quantify. The danger with these types of questions is that not every learner answers it and only those with a strong opinion decide to share it. This means that one person’s view about the length of a breakout or the difficulty of an activity may end up being given too much, or too little, weight.
   
   
5. Consider the next step. There is no point in doing any kind of evaluation if no further action will be taken. Schedule time to review the evaluation and time to act on the results of it. Decide in advance when program reviews will take place and the cadence of the revisions. Developing a plan is key for implementing changes to better facilitate learning in the classroom.

What Do Virtual Learning Experts® Need To Know About Evaluating Hybrid and Virtual Learning?

Evaluating training programs and their effectiveness for the learners and the organization is one thing, but it’s also important to assess the skills and behaviors demonstrated by the instructional team. In addition to understanding how the program will be evaluated, Virtual Learning Experts® need to know how they are being evaluated and assessed too.

1. Virtual classroom designers need to know how the program will be evaluated so that they can begin with the end in mind. This involves making sure the program is designed to achieve its aim. They also need to know how the learners will be assessed, so that they can ensure the training is aligned with the assessment criteria. In addition, designers need to know how they themselves will be assessed. They need to know what’s expected of them and what standards they should consistently be working towards.

2. Virtual classroom facilitators need to know how the learners will be assessed so that they can ensure enough time is devoted to the right topics. They need to understand how learners will be assessed so that they can provide feedback and support to set the learners up for success. To properly improve learning facilitation, this role also needs to know what’s expected of them and the standards of facilitation expected by the organization.

3. Virtual classroom producers

   need to know the logistics behind any evaluation or assessment so that they can support learners in accessing it. They need to be able to work with the facilitator to ensure the learning objectives are met and act as the learner's advocate as needed. Producers also need to know what’s expected of them so that they are consistently reaching, or exceeding, the standards expected by the organization.

4. Virtual Learning Coaches

   need to know how to assess the skills demonstrated by designers, facilitators, and producers. They need to know what “good” looks like and what “great” looks like for their organization. They need to be able to use the rubrics to ensure consistent and fair evaluations throughout the team. This is integral to improved performance and the development of virtual classroom strategies
   

   Coaches also need to be able to evaluate their own skills too. Coaches need coaches too! Ideally, Virtual Learning Coaches® will work with another coach to help them see where they can improve their practice. However, this isn’t always possible, so coaches also need to be able to self-evaluate. Catherine Lombardozzi explains how to use the mirror as an evaluation tool in this blog post.

Learn More About Evaluating Hybrid and Virtual Learning and Other Teacher Training Tools

Join us to learn how to gather the right data and use it to evaluate the effectiveness of your hybrid and virtual training programs.

Are you wondering how to evaluate hybrid and virtual learning? We can help with our Trends in Virtual Training - Expert Seminar Series workshop Evaluating Hybrid and Virtual Learning. And there are 11 more trending workshop topics for you to explore!

Purchase 5 workshops and the 6th workshop is free. Click here to learn more about the entire series.