Discover the Key to Effective L&D Measurement: Start with Business Objectives
In our previous discussion, we delved into the “streetlight effect”—the metaphorical tendency to measure what’s easiest rather than what’s most relevant. Learning and Development (L&D) often falls victim to this trap, assessing results where it’s convenient instead of where it truly matters.
Remember, the ultimate aim behind our measurement efforts is diverse. Whether it’s for continual improvement, compliance, or assessing impact (including ROI), it’s vital to define your objective before diving into metrics!
So, how do we break free from the constraints of the streetlight effect in L&D? The first move is to flip the process: instead of creating a training program and then questioning, “How do we evaluate its success?”, start with the desired outcomes. Determine the business goals you want to accomplish and let that shape your training design and measurement strategy.
Developing a Reverse Data Strategy
Although beginning with business objectives might seem like common sense, it represents a significant pivot in approach. Alarmingly, only about 4% of organizations claim to design learning initiatives based on predefined metrics from the outset. What happens to the other 96%? They often base programs on perceived needs or requests, deploy the training, and then consider evaluation afterward—if they even do so at all. By failing to integrate measurement during the design phase, L&D teams often result in minimal assessments that rely heavily on superficial post-hoc metrics.
Starting from identified business goals clarifies what success looks like. For instance, if the company’s target is to lower safety incidents by 20%, that becomes your guiding star. From there, you can backtrack:
- Identify who can effectively contribute to reducing safety incidents—target the audience that will make the most significant impact.
- Determine the necessary behavior changes to achieve that 20% reduction.
- Identify which employees need to adopt these behaviors.
- Assess current barriers (skill gaps, knowledge deficits, motivation, or process hindrances).
- Only then, decide if training is the solution. If so, tailor the program to target those key behaviors.
- It’s crucial to outline Key Performance Indicators (KPIs) from the start—in this case, the safety incident rate—and devise a plan to monitor it. This might involve establishing baseline data, and then comparing outcomes post-training (potentially against a control group) to observe changes. Additionally, consider on-the-job observations to track adherence to new safety protocols.
This method, known as “backward design,” guarantees that training efforts are purposeful. It might even expose instances where training isn’t the proper answer, such as when poor processes, inadequate tools, or incorrect incentive structures are at play. By anchoring your efforts to organizational goals and conducting a thorough needs analysis, L&D teams can avoid misdirecting resources on programs that don’t address the core issues.
Ensuring Business Alignment
Research from the Association for Talent Development found that only 43% of talent development professionals believe their business and learning objectives are aligned.
When L&D designs initiatives in alignment with business goals, measuring effectiveness becomes much clearer. Clear targets (KPIs and desired behavior changes) can be established, and relevant data gathered effectively. Instead of searching aimlessly in the dark, you possess a clear roadmap directing you to where the real challenges lie.
In doing so, over time, L&D can foster credibility. Business leaders recognize that L&D is prioritizing outcomes that matter to them—like sales enhancements, quality improvements, or reduced turnover—rather than superficial metrics, such as attendance rates. When a training program fails to deliver expected results, it offers a learning opportunity rather than a reason to hide behind vanity metrics.
The essence of measurement is to unveil what succeeds and what fails, not just to validate results. When L&D concentrates on the aftermath of learning events to assure desired outcomes, it evolves from being perceived as a mere cost center to a strategic partner illuminating data-rich insights useful for business decision-making.
Guiding Frameworks and Models for L&D Measurement: Kirkpatrick, ROI, and LTEM
Fortunately, L&D professionals don’t have to navigate this journey in obscurity. There are established models and frameworks for training evaluation that provide direction—like signposts illuminating our path. Three leading models are: Kirkpatrick’s Four Levels, Phillips’ ROI Model, and the Learning-Transfer Evaluation Model (LTEM). Each offers unique insights into what to measure, encouraging us to look beyond surface-level metrics.
Kirkpatrick’s four levels of evaluation is widely recognized but often misapplied in corporate environments. Many start with level 1 assessments and get bogged down, focusing primarily on immediate feedback such as reaction rather than progressing through the subsequent levels.
Jack Phillips further enhanced this framework by introducing a fifth level: ROI. This level asks whether the training was financially worthwhile. The Phillips model assesses the financial benefits derived from training compared to its costs, producing an ROI percentage. For example, if a leadership development program costs $100,000 and yields an additional $300,000 in productivity, the ROI would be a robust 200%. This resonates well with executives as it speaks their financial dialect.
Calculating ROI can be complex: isolating the training’s financial impact involves certain assumptions. Phillips promotes methods like translating improvement metrics into monetary terms and soliciting participant estimates on training’s effectiveness, while also accounting for optimism bias. The key takeaway remains that our focus should be on real outcomes, not merely activities.
Both Kirkpatrick and Phillips stress a fundamental truth: training evaluation isn’t complete until examining job impact and organizational outcomes. In simpler terms: did behavior actually change, and did it matter to the business?
Exploring the Learning-Transfer Evaluation Model
In my recent practice, I’ve turned to a new model, the Learning-Transfer Evaluation Model (LTEM) developed by Will Thalheimer. This eight-tier framework addresses common shortcomings in measurement practices by focusing on learning transfer—essentially assessing if individuals are applying what they’ve learned.
The initial tiers of LTEM (1 and 2) deal with attendance and engagement—did participants show up and engage with the learning material? For engagement assessment, we track physical actions, emotional responses, and cognitive challenges. Tiers 3 through 6 look at retention and real-world skill application, yet these assessments typically occur in a training environment, which is less impactful. Tier 7, however, marks the pivotal moment where we evaluate whether learners are successfully applying skills on the job.
Intentional Behavior Change
LTEM tier 7 focuses on actual behavior changes in the workplace, akin to Kirkpatrick’s level 3 but with an emphasis on assessing performance in real environments. Finally, tier 8 explores the broader organizational impact—essentially evaluating how improved performance affects business outcomes and even influences colleagues or customers.
The rationale behind adopting LTEM lies in its detailed perspective on what generates value from training. Coupled with the backward design framework, this model offers practical guidance for various L&D roles that aspire to effectuate meaningful change. Stay tuned for more insights in our upcoming article.
Impact Isolation: Effective L&D Measurement
One of the primary obstacles mentioned in recent surveys by the Association for Talent Development is that L&D practitioners often find it challenging to isolate training impacts. This is a valid concern. Hence, I advocate for measuring and designing solutions by reversing the process: commence with business goals and intended outcomes, followed by defining supporting performance objectives, identifying key audiences, and then mapping out necessary behavior changes. Without behavioral shifts, there can be no tangible impact.
Regardless of the measurement frameworks or models employed, implementing a backward strategy focusing on business objectives will facilitate clearer isolation of learning impacts. Upcoming discussions will also address how AI can provide invaluable assistance in this endeavor and how various L&D roles can benefit from this technological advancement.
References:
[1] Measuring Learning’s Impact
[2] ATD Research: Organizations Struggle With Measuring the Impact of Training
[3] Model vs Framework: Understand How Each of Them Work
[4] ROI Methodology
[5] What Role Does TDRp Play in the Measurement Space?
[6] Beyond Kirkpatrick: 3 Approaches to Evaluating eLearning
[7] Measuring Learning: Asking The Right Questions
