Measuring your Digital Learning ROI – How to do it?

A lot has been mentioned about the digital transformation that organizations are currently undergoing. According to a PwC survey, annual digital revenue adds up to ~$500 billion over the next 5 years. Enabling digital technologies in select industries is expected to save $421 billion every year for the next 5 years. Interestingly, the survey which includes responses from 2,000 people across 26 countries notes that the biggest challenge in digital transformation is people related.

As a $300 billion industry worldwide, learning and development has long been touted as a key beneficiary of digital technologies and how digital learning can disrupt traditional learning. Given that digital learning is still relatively new, a risk associated with digital learning are the unknown variables that organizations can miss out when factoring their costs for implementing digital learning. In such a context, organizations would stand to gain if they consider various approaches to evaluating effectiveness or the return of investment for their digital training objective.

Measuring ROI for Digital Learning

In general, the ROI for learning is debatable given that the impact of learning can be quantified in many ways. While the calculation is no different and simplistically looks at the gain divided by cost, sometimes it becomes difficult to pinpoint exactly what is the gain accrued from training. Often the key question to consider is the time within which to calculate the ROI.

As a start, looking at traditional ROI approaches to learning is often a reference for digital learning. The Kirkpatrick model, for instance is a great framework to use. The model consists of measuring 4 outcomes:

Behavior (L3) – the transfer of knowledge, skills, and/or attitudes from classroom to the job

Results (L4) – the final results that occurred because of the training program

While the Kirkpatrick model is used extensively for measuring traditional training initiatives, it may require tweaks or modifications when it is employed for evaluating digital learning. For instance, while measuring reaction can be done effectively for a two-day training program, in the case of digital learning where a learner has access to learning content over a longer period, measuring reaction would be prudent to capture at multiple instances during that time period rather than at the end of the time period.

The same approach would apply to measuring learning (L2), behavior change (L3) when it comes to digital learning. As we can see, the use of digital learning enables organizations to calculate ROI across multiple time periods rather than only at the end of the period, as is the case with traditional learning.

Another simple but extremely effective approach to measuring ROI comes from none other than Andy Grove, Intel’s famous CEO. Andy viewed training as a key lever to improve workforce productivity and focused on how training enabled organizations to save time by improving productivity. In the book by Ben Horowitz entitled The Hard Thing About Hard Things, Andy is quoted saying

“Training is, quite simply, one of the highest-leverage activities a manager can perform. Consider for a moment the possibility of your putting on a series of four lectures for members of your department. Let’s count on three hours of preparation for each hour of course time – twelve hours of work in total. Say that you have ten students in your class.

Next year they will work a total of about twenty thousand hours for your organization. If your training efforts result in a one percent improvement in your subordinates’ performance, your company will gain the equivalent of two hundred hours of work as the result of the expenditure of your twelve hours.”

Calculating your Digital Learning ROI

As mentioned previously, calculating ROI for learning is a function of the following cost elements

Money: The most obvious and tangible cost component is money. This involves direct cost expenditures such as
a. Buying or creating 3rd party learning content
b. Investing in a digital learning platform
c. Customization costs

If organizations develop their own learning content or platform, then the cost saved would automatically appear under the People cost category under time invested and salary expenses. Additional outflow of money would involve investment in technology infrastructure –on premise or on cloud servers, among other things

People: People costs are sometimes not as straightforward to factor but often include:
a. Time invested in evaluating digital learning vendors
b. Cost to upskill key people to implement digital learning

Time: As microlearning and bite sized learning take more prominence in digital learning, we will get a better understanding on the impact that time plays for both learners and organizations on the effectiveness of learning. Often the most underestimated but ultimately the most important cost element continues to be the time invested in implementing digital learning. This includes the time invested by:
a. The learners
b. Project manager
c. Support team and
d. Business sponsor

While the cost elements fall within these three elements, the ability to find value from digital learning can come from many more avenues. We can look at evaluating ROI from digital learning via quantitative and qualitative methods.

While the cost elements fall within these three elements, the ability to find value from digital learning can come from many more avenues. We can look at evaluating ROI from digital learning via quantitative and qualitative methods.

Quantitative ROI

Time saved: Often in digital learning, time invested can translate to time saved. For instance, conducting product training via digital learning over a 3-month period versus a traditional two-day program for a team of 1,000 sales reps can translate into the following time and productivity savings:

Calculating time saved during a digital learning initiative for sales

Time created: Not easy to identify, but often an effective digital learning solution can also leverage time bursts that were previously not used for learning nor for anything productive related to work. For instance, imagine waiting for a customer meeting for 5 minutes. While previously this 5-minute time slot could not be leveraged for traditional learning, in a digital learning scenario, 5 minutes is ample time to refer to relevant microlearning content that is germane to the customer meeting. If you extrapolate this situation for the same 1,000 salesforce of a company, that can create as many as 5,000 minutes of training time generated in a month from the same digital learning solution without additional investments.

Travel Costs: Leveraging digital can imply reducing travel costs for both learners and possibly for trainers as well. Assuming 2 trainers are used for the product training initiative for the 1,000-sales force strong company, deploying digital learning can reduce or eliminate travel expenses anywhere between 30% to 100%.

Qualitative ROI

Analytics: If utilized well, analytics can tell a powerful story to business executives on the effectiveness of digital learning. To put this in context, analytics can showcase three powerful storylines:

Utilization of learning: To start simple, it’s important to understand how many learners are utilizing digital learning as a percentage of total. More nuanced measurements can include:

Number of hours or minutes per month or week

Most active period for learning

iii.Number of hours spent by learners post the launch of new modules

Progression: As driving digital learning induces a self-directed approach to learning, organizations can view progression of learners in real time to identify progressive learners and laggards. This level of identification would be less clear when implementing a traditional learning approach.

Skill gaps: A crucial element to evaluate as a result of measuring usage and progression in a digital learning initiative is the analysis of retention and application of learning. While this is similar to L2 and L3 of the Kirkpatrick model, the level of data can be more authentic and detailed for digital learning. For instance, imagine knowing what are the gaps in retention in product knowledge for your sales force based on various demographics such as tenure, region, product features among other things.

Learner Feedback: As mentioned earlier, unlike the traditional L1 (or zero-hour feedback) from the Kirkpatrick model, learner feedback can be measured continuously in digital learning for both quantitative and qualitative data points. This allows organizations to see how data is trending over an extended period – something that may be harder in traditional learning.

Learner Engagement: One of the biggest drawbacks of traditional learning is to sustain learner engagement or for that matter even measure it. With digital learning, that becomes much easier to measure. Organizations can view how often are learners viewing learning content, build leaderboards to create competition, allow users to generate content and much more.

Conclusion

As we can see, measuring ROI in digital learning can be more quantifiable than leveraging traditional learning. Furthermore, digital learning can also produce a wealth of data that can be useful for showcasing learning effectiveness in terms of analytics, engagement and feedback from learners.

Hence calculating ROI implies taking a comprehensive approach using both quantitative and qualitative data points. Given how digital learning is a continuous process, ROI calculation can also become an ongoing practice allowing organizations to calibrate their digital learning initiatives. This can help them to double down where they are seeing more value (more time saved, more utilization and more engagement) and phase out where such value is elusive.