Are You Making a Real Impact? Discover What Truly Matters
Perhaps you’ve heard a version of this classic tale:
Under the glow of a streetlamp on a quiet, moonlit night, a somewhat tipsy figure bends over, rummaging through the shadows. A passerby inquires, “What are you looking for?” Amidst a sigh, the figure replies, “My key.” After a few moments of searching, the passerby asks, “Are you confident you lost it here?” The man shakes his head, revealing, “No, I lost it in the park.” Puzzled, the stranger questions, “Then why search here?” The man points to the bright circle of light and says, “Because it’s easier to see.”
While we might chuckle at the absurdity, this scenario, often dubbed the streetlight effect or the drunkard’s search, highlights a common human tendency: seeking answers in convenient but not necessarily relevant data.
L&D’s Own Streetlight Effect
In the realm of Learning and Development (L&D), a similar dynamic plays out. Faced with the critical question, “Are we truly making a difference?”, many professionals find themselves sifting through the readily available metrics: LMS reports, course completion rates, and satisfaction surveys. This tendency isn’t purely for efficiency; it’s about comfort. The vital indicators of impact—hidden within performance dashboards, sales data, or customer satisfaction metrics—tend to be overlooked because they’re more challenging to access.
Consequently, L&D teams often remain in the well-lit zone, focused on metrics like attendance and quiz scores. While this feels safe and satisfying, it only reflects the streetlight effect: measuring what’s simple rather than what’s significant.
The Challenge: Easy Metrics vs. True Impact
This inclination to “search where the light is bright” leads to several pitfalls in measuring L&D success. Consider some common statements:
“We had 500 participants in the workshop, and 95% would recommend it!” or “Our LMS reports show 1,200 course completions this quarter, totaling 600 hours of training!”
Such metrics, while shiny and easy to collect, don’t genuinely reflect whether employees enhanced their skills or if the organization benefited from the training. In many cases, businesses might not interpret those 600 hours of training as valuable investment.
A study revealed that many businesses “over-rely on basic metrics like completion rates and satisfaction surveys.” These are the types of data that shine bright under L&D’s streetlight: easily measurable, automated, and reassuring—much like the comforting glow of a streetlamp.
The Association for Talent Development found that only 43% of talent development professionals believe their learning and business goals are aligned.
If alignment is lacking, we must ask ourselves: Are we truly focusing on what matters?
What Are We Overlooking?
One of my favorite inquiries when addressing business challenges or opportunities is: “What are we not considering?”
While it may seem costly to slow down and ask these questions, relying solely on easy metrics can lead to even greater costs. By remaining focused on straightforward numbers, organizations often miss vital insights hidden in less visible areas. As one report pointed out, many companies mistakenly assume that if participants complete training and express satisfaction, the training must be effective. This assumption—a dangerous one indeed—mistakenly equates completion with success.
In truth, course completion and satisfaction ratings don’t guarantee learning, behavior change, or tangible results. A learner may rate a course highly due to its entertainment value but may not apply any new knowledge in their role. Similarly, achieving a 100% completion rate for mandatory training doesn’t necessarily translate to fewer safety incidents or improved sales metrics. Under the bright lights of completion rates and average scores, the failures to initiate real change remain obscured.
Is It Just Me?
You’re far from alone in this. With 25 years of experience across various organizations, both large and small, I have consistently observed these patterns. In fact, most L&D teams struggle to measure deeper impacts. Industry surveys reveal that 43% of organizations report no Level 4 measurement at all, which pertains to evaluating true results and business impacts.
Why We Remain in the Light: Barriers to Effective Measurement
Research and practical experience reveal a striking commonality: L&D teams recognize the importance of measuring what matters, yet they often find themselves constrained. So, why the disconnect?
It’s not due to a lack of interest or effort. In fact, 91% of companies acknowledge the necessity of measuring the impact of learning beyond the basics. The intent is evident. However, several underlying barriers continue to restrict L&D from moving beyond surface-level metrics:
- Not Knowing Where to Start: Identifying how to measure behavioral changes or business outcomes can feel daunting. Often, teams lack a clear strategy. Many find it easier to rely on familiar tasks like collecting course feedback and test scores rather than navigate untested analytical avenues. Remember, starting where you are is perfectly acceptable; incremental progress is preferable to waiting for optimal conditions.
- Difficulties with Data Access and Integration: Accessing metrics from areas that matter often involves gathering data from outside the L&D confines. This might mean pulling information from sales systems, quality assessments, or HR performance records. However, many L&D teams face challenges due to data residing in disparate systems, governed by different departments. Access issues can frequently hinder effective learning measurement.
- Lack of Business Alignment and Stakeholder Support: Measuring meaningful impact often necessitates collaboration across various business layers. You may require managers to report behavior changes or execs to prioritize measurement efforts. Unfortunately, convincing stakeholders to prioritize such in-depth evaluation can be a challenge. Many are content as long as employees fulfill training requirements, leaving L&D without the necessary resources or support to probe deeper.
- Limited Skills and Confidence in Analytics: Not every L&D professional is a data analyst, and they don’t need to be statistical experts. Yet, achieving today’s learning outcomes often demands varied skills, including robust data analysis. Many teams may lack the competence or tools needed for thorough evaluations, contributing to reluctance in moving beyond basic reporting.
- The Complex Nature of Behavior Change: Even with available data and analytical skills, measuring behavior change and its relation to training effectiveness is intricate. Behavior change can evolve over time and is influenced by numerous factors beyond training. As a result, organizations may hesitantly avoid deep-diving into this subject.
These barriers often keep L&D measurement confined to easily accessible metrics. Yet, avoiding deeper measurement has consequences. Lacking meaningful evaluation risks fostering a “putting content out there and hoping for the best” mentality. Alarmingly, 69% of companies identify the inability to measure impact as a critical barrier to achieving essential learning outcomes.
Thus, not measuring impact translates into more than just a measurement dilemma; it escalates into a business-wide concern.
This reality impairs L&D’s ability to align with strategic objectives, subsequently limiting its capability to demonstrate and enhance its organizational value.
How to Progress: Focus on What Matters
In the forthcoming articles in this series, we’ll delve into methods to transition from the safety of familiar metrics to uncovering where true impact lies. We’ll explore various measurement and evaluation models beyond the well-known Kirkpatrick framework. Additionally, we’ll investigate how leveraging AI can empower L&D teams to amplify their reach and effectively measure what truly matters.
References:
[1] Understanding the Streetlight Effect
[2] Measuring L&D’s Effectiveness: Key Reports
