Thousands of Flashlights: How AI lights up dark corners
Two previous articles in this series investigated the effects of streetlights. That is when you tend to look for something useful (under the street lights) rather than where it is (dark park). In the case of learning and development (L&D), it means measuring easily available metrics in our control (course completion, time spent training, and satisfaction). This last article shows how to lighten up at dark corners to measure the impact of learning and think of AI as a force multiplier in L&D analysis.
Why is it difficult to measure the impact of learning?
Consistent with some research and my own experience, the latest ATD research on the future of learning assessment revealed the same barriers and challenges. [1]:
Lack of time and resources access to data Lack of lack of skills Lack of buy-in and support from stakeholders
Does it sound familiar? It’s no wonder the L&D stays in the bright areas of the LMS. To measure the impact on your job, you need to break out of the LMS bubble and work with business, IT, talent acquisition and more. Dark corners require many flashlights. To date, scalability due to lack of time and resources appears to be one of the biggest barriers.
Technology doesn’t remove all barriers. Culture, lack of clarity, broken processes, unclear goals and responsibilities, lack of accountability, etc., must be addressed by humans before artificial intelligence (AI) can be useful.
The ability to quickly and intelligently illuminate light at once in many places to see the full picture of impact is the promise of data analytics, AI, and automation.
Six ways AI as a force multiplier can help you measure and evaluate
1. Strategic support
Prioritization, trade-off analysis, backward design, and ROI calculations are some of the examples where automation and AI can provide guidance on what it takes to begin with and how it should measure success.
2. Design support
Even before measurement and evaluation, AI can be used to help, for example, write evaluations. For learning designers, I have created an AI bot that analyzes your assessment questions and provides detailed scoring, suggestions and feedback on the approach. These assistants are now evolving into agents with the ability to do things as well as act.
3. Turning satisfaction surveys into performance-focused questions
Turning background satisfaction surveys into performance-focused questions that generate actionable data insights is another area where AI can help. Another AI service is trained in a learning transfer assessment model (LTEM) and performance-focused research question design to help create more practical data [2].
4. Analyze data at large scale and depth
AI in L&D measurements helps you collect and analyze data at previously unrealistic scales and depths. If human analysts can struggle to correlate their training data with six different performance metrics spread across three systems, AI-driven tools can crunch those numbers in seconds and spot patterns. For example, AI can track learning data over time along with performance metrics to identify correlations, compare groups (who did not) and even analyze qualitative data (such as open-ended research responses and work product samples) to see if learners are applying skills. [3].
5. Lack of time and resources
Analyze open-text responses, real-time chat interactions, or small breakout group conversations to summarise insights, find patterns, categorize responses, predict emotions, etc.? It’s resolved.
6. Immersive dialogue and embedded measurements
The February 2025 ATD TechKnowledge Conference shared a prototype of a 3D adventure that allows users to interview interested individuals based on a specific rubric of best practices. AI characters interacted in real time and had short and long term memories. They shared facts about the world, but they also knew each other. Finally, the AI coach provided a detailed analysis of the interview. All of this was built by me within a month. My prediction is that this type of immersive activity will soon be available on all decent learning platforms.
One 2025 industry report allows Advanced AI to enable a more sophisticated approach to linking learning and performance, while AI-powered analytics can assess understanding, applications, behavioral changes, and more, which are the “true drivers of business performance.” [3]. This means that AI is not fascinated by streetlights. It is actively looking for a shocking glow in the darkness.
Predictive analytics that provides actionable insights
Furthermore, AI can be predictively prescribed. Through predictive analytics, AI may highlight which employees are most likely to benefit from a particular training (and therefore L&D can better target interventions). It also helps to identify whether training is revealing performance issues that could be helpful, essentially warning L&D of the need before business is required. In our ratiophor, AI not only shines light where the key is, but can even predict where it is first visible (“based on past patterns, the key is usually dropped near a park bench”).
And finally, privacy and ethics cannot be ignored. Squeezing the light everywhere does not mean spying on employees or violating trust. The goal is to lighten the impact rather than invade privacy.
We have the technology that truly measures what we have always cared about. Actual behavioral change and business outcomes are scalable, real-time ways. Think of AI as a multiple of the power of its impact on the New World, not a threat to work against old jobs.
Bright future: Measuring important things in every L&D role
Descending from the narrow circle of streetlights and delving into the broader, brighter landscape of measurements, is not just a respectfulness, but the future of the L&D. And it requires a cultural change touching on all roles in the L&D field:
For educational designers
This means designing with measurement in mind. Using models such as LTEM, learning solutions include opportunities to demonstrate your application.
For L&D Program Managers and Facilitators
It is to enhance and follow up work learning. You can also partner with a Line Manager to gather feedback on behavioral changes, or set up post-training touchpoints (such as refreshes and coaching sessions) to provide forwarding and insights about progress. Instead of declaring success when the class ends, you will see that your role is spreading to the workplace. You can teach your managers on how to support new behaviors, make light measurements such as sampling work output and focus groups, and hear people apply (or not) training.
For L&D leaders
This is about strategy and culture. Leading your fees when adjusting your learning to business goals. I support tools and resources that allow teams to measure what is important (probably invest in LRS, or analytic talent, or AI platforms). They will also fall into you to educate their stakeholders. We look forward to executives that L&D will report on business outcomes as well as activities and fulfill their promises. Why not use the measurement rubric to prioritize project requests that stakeholders are willing to collaborate on measuring their actual impact?
For learning analysts or data scientists
Analytics and facility skills using AI tools can help you translate raw data into meaningful stories. We experiment with a variety of methods (A/B tests such as training, predictive modeling) to truly understand causality, as well as correlations.
Conclusion: AI is a force multiplier
Ultimately, avoiding the streetlight effect with L&D measurements means having the courage to seek truth, even in places that are vague and difficult. It means trading the instant comfort of simple metrics for a more rewarding payoff of meaningful metrics. Yes, measuring how new software training has affected productivity is more difficult than counting the number of people opening training videos. But which one would you bring to your CEO? Do you actually tell us if your training was successful?
References:
[1] The future of learning assessment and measurement of impact: Improve your skills and addressing challenges
[2] The effectiveness of learners’ research and learning is Will Talheimer
[3] Measure the important things: Connect learning outcomes with AI to business outcomes
