Evaluating and Assessing Modern Blended Learning Outcomes

In discussing the concept of “blended learning,” we have previously stated that “modern blended learning is selecting the most appropriate delivery technology based on learning objectives, with consideration of time, place, and space.” So it is that, as we consider how to assess a blended learning solution, we need to look at that statement and look at the objectives that required a training solution in the first place—finis origine pendet— “the end depends upon the beginning.”

When a blended solution is implemented, it is directed at providing learners with the opportunity to master/complete a particular set of individual learning objectives. To assess such a training approach, many will stop there and seek to assess against the individual objectives, absent any consideration of the delivery method or specific organizational outcome that drove the individual learning objective treatment in time, place, or space.

To illustrate, let’s look at an example:

Mel is an insurance adjuster for Magnanimous Insurance Company, Inc. Magnanimous has just released a mobile technology solution that allows adjusters in the field to fill out the requisite claim documents and append pictures and related media to expedite the claim for the customer. Magnanimous implemented this technology to reduce error and time to conduct a field damage assessment, thereby reducing its costs. To support the roll-out of this technology, Magnanimous provided self-paced (time) training online via mobile tablet (place), to allow adjusters to practice in actual field situations (space) while using the old method to document the claim. In addition, they held weekly live virtual seminars to allow debrief of feedback from the field and to solicit any input on making the mobile solution better.

QUESTION: How do we assess this training outcome?

ANSWER: Look at the desired organizational outcome(s), and then evaluate how that objective is supported by learner outcomes. Magnanimous Insurance wants to streamline the submission and documentation of claims— doing so lowers costs—but as they are in a highly regulated industry, Magnanimous also needs the documentation to be complete and correctly submitted. Using the old system, the error rate was 4.7%. When stated this way, the assessment and evaluation of the learner and the blended training solution becomes fairly apparent—use the operation of the mobile application to evaluate actual performance by Mel and his peers and compare the error rates with those from the old way of submitting claims. Of note, you’ll NOT see any of the affective “smile sheet” kind of data that trainers often rely on—frankly, the only feedback Magnanimous might want from its trainees is the user feedback on layout and ease of use to improve its mobile application and claims adjustment process, along with learning-environmental data (as this is self-paced content, things like connectivity to the app, server down time, user latency in the lesson, usage by individual and role, etc. Other learning environments bring their own data sources of interest). User feedback (whether affective “smile sheet” kind of data or more quantitative user data, is important: the point is that it must be used within the limits of what it can tell a program evaluator and provided context and relationships before collection.

There's one other important note in this example: the assessment and evaluation approach needs to be considered long before the training solution is implemented. Like most traditional development models (such as ADDIE), a learner assessment (and we would add organizational evaluation) in the training solution must be planned and designed before actual implementation. The program evaluator needs to build measures and measurement protocol BEFORE data starts coming in; otherwise, there is a (pernicious) inclination to “read the data” and develop conclusions that may be unsupported by the data (this is where things like confirmation bias LOVE to play).

“OK, I get it,” you might be thinking, “But how is blended assessment any different?” Great question!

Blended Assessment is MORE Capable

The key to understanding the difference is in the varied learning environment and capabilities at your command in a blended solution. Rather than relying on delivery in the same treatment method(s) used to deliver the training content, you have largely unfettered use of all blended methodologies to assess, and, in addition, can draw from latent data sources (say, for example, mis-filings of insurance claims in the mobile application or user latency in filling out electronic forms) to determine learner progress in achieving desired outcomes. Moreover, with many blended treatments and environments, there is an inherent level of fidelity to real world performance that a traditional method of instruction or assessment can’t always provide.

To conclude, the assessment of the learner and the learning environment can be integrated with an evaluation of the blended training solution to provide insight into both learner progress and the achievement of organizational outcomes. With a variety of technologies, environments, and strategies to assess the learner, don’t feel constrained to stick with the old multiple-choice-test-and-effective-instrument solution—leverage the capabilities of the modern blended classroom to take advantage of passive data collection and tools!

To improve your assessment in blended learning, InSync’s Evaluating Hybrid & Virtual Learning Workshop equips you to find actionable data that can truly inform your learning approach. We cut through the complexity to give you the tools to really understand what makes hybrid and virtual learning tick.