Everyone took the course. No one can open the tool.
I once participated in a review where a client viewed a dashboard that showed a 96% course completion rate for their AI rollout. Everyone nodded. The numbers were green. Then someone from operations asked, almost as an aside, if anyone on the floor was actually using the tool. The room fell silent as no one was measuring it, and the honest answer turned out to be no. The gap between the course being completed and the tool being deployed in use is something no one puts on their dashboard. And that was the sole purpose of this project.
Completion Action Attendance rather than modification
A gym membership can help you think about this. At the gym, you can see that you have scanned your card 11 times in the last month. I don’t know if you are stronger. Check-ins are real, easy to count, and almost completely separate from what you actually signed up for. To complete the course, scan your card. A passed quiz will be a little better. Both don’t tell you if anything has changed where the work is done.
There is a useful framework for this, created by a training researcher named Donald Kirkpatrick, that shows four levels of assessment. Level 1 is whether people liked the training. Level 2 is based on whether you have learned the content and can be checked with a quiz. The important thing here is level 3 (does your behavior actually change at work?), but this is a level that almost no one measures. Above that, there is also Level 4, which is the actual results the business was looking for. The reason I focus on Level 3 behavior is because it is a leading indicator of those outcomes. That means fewer callbacks, fewer reworks, and fewer exact mistakes where you bought a tool to stop it. As your behavior changes at work, your floor number will follow. You don’t need academic vocabulary to do this successfully. You need to know that completion and quiz scores are on the easy level, and the important stuff is on a higher level and is hard to see.
What adoption actually looks like
Changes in behavior leave a mark if you know where to look. Weeks after the training is over, are the tools being opened at the intended workflow steps without anyone alerting people? Are employees bringing it up themselves, asking questions about edge cases, and telling you where they’re missing? These questions only come from people who are actually using something. When the people you train start talking about where the tools are falling, it may sound like a complaint, but what you’re actually hearing are the people who have incorporated that tool into their daily routine enough to find that edge. That’s the sound of adoption.
Usage that persists beyond the first two weeks is the signal I trust the most. Immediately after training, almost everything comes under impact. The question is whether the curve flattens out in real numbers or returns to zero as novelty and manager attention move forward.
Measure without being monitored
The wrong way to measure adoption is to log every keystroke and track individual workers against each other. The moment people feel like they’re being watched, you’ve poisoned what you’re trying to measure, filling the floor with people who only use tools when they think someone is watching.
Aggregated signals are sufficient, and most of them can be counted without singling out anyone. How often the tool is opened at the appropriate workflow step. Counts by team, not by name. The support tickets and callbacks that the tool was supposed to reduce actually decreased by a quarter. A short, honest monthly check-in to ask what’s working and what’s not. A quiet word to a few people leading on the front lines about whether they are witnessing the action on the ground. All of them are read at a team level, not an individual level. You’re trying to learn if there’s been a change in work, rather than building a case file on a specific person. The lighter your hand, the more accurate your reading will be.
Set this expectation before you start
This is the part that needs to be done before the engagement, not after. When a buyer asks me to build an AI training for them, done is usually the number they have in mind as the finish line. So I tell them early on that completion is not the deliverable, it’s the change in behavior that matters, and that we’re not just looking at whether everyone clicked by the deadline, we’re looking at usage on the floor 60 and 90 days later. Some habits become established by then, while others take longer, so I treat these checkpoints as readings on hiring (or other) trends rather than final grades.
This conversation changes the shape of the entire project. Because a course made to be completed and a course made to be used are not the same course. Completion is a number that came in expecting an upward report, so a few buyers will initially push back. But once most people accept it, they’d rather know the truth than hide away a tool that no one touches and achieve a 96% completion rate. They wanted us to use that tool. They just weren’t told that completion and use are two different things and that they had to choose one or the other first. So I tell them that and measure the right things from day one.
Share with
