Guide
Measuring Whether AI Is Actually Helping, Without a Surveillance Dashboard
You need to know if the tools are landing. You do not need a leaderboard that ranks your engineers by keystroke. Here is how I get the signal without the surveillance.
The pressure to measure AI adoption usually arrives from above, and the fastest way to satisfy it is also the worst: stand up a dashboard that counts each engineer's tool usage and sort it high to low. It feels like measurement. What it actually does is teach your team that the number is the target, and a number that is a target stops telling you anything true. Here is the approach I use instead, aimed at the one question that matters, which is whether the work is getting better.
Decide what you're actually trying to learn
Separate three questions people mash together:
- Is the habit happening? A yes/no per unit of work, not a ranking.
- Is the thinking any good? A quality judgment on an artifact, aimed at coaching.
- Is anyone stuck? A support signal, aimed at helping the person early.
None of those three is "who used the tool the most." Usage counts feel like all three and answer none of them. Once you name the real questions, the surveillance dashboard stops looking like the answer.
Measure the artifact, not the activity
Pick one AI habit that leaves something you can read. Mine is an implementation plan drafted with AI and attached to the ticket. That gives me a clean yes/no on the habit (the file is there or it isn't) and, more importantly, an artifact whose quality I can actually assess. I periodically have AI evaluate each plan on the strength of the thinking, and every engineer gets their own feedback. The signal I care about most is a trend: is the same engineer's tenth plan stronger than their first? That is a measure of learning, and it is invisible to any usage counter.
Pick the one usage number worth an alert
There is exactly one raw usage signal I put an alarm on, and it isn't "too little" or "too much." It's lots of sessions with nothing accepted. High activity that produces nothing usable means a person is spinning their wheels with the tool, and that is a coaching moment, not a policing one. Everything else in the usage data is context I glance at to know where to look, never a scoreboard I publish. Spend anomalies are a budget conversation. Wheel-spinning is a mentoring conversation. Neither is a ranking.
Make the data flow to the person first
If AI usage is going to be visible to leadership, and it is, the cheapest trust you will ever buy is making it visible to the individual first. When I built a place for engineers to see their own AI spend and usage, I labeled estimated and billed numbers separately and gave each person their own view before any of it rolled up to me. Transparency that only flows upward isn't transparency, it's surveillance with better branding. Flip the direction and the same data becomes something people trust instead of something they hide from.
Say out loud what you won't do
Tell the team the rules of the measurement, because the fear is the real adoption killer. No leaderboard. No ranking people by usage. No individual number in a performance review. The measurement exists to find where the tools help and where someone is stuck, full stop. When people believe that, they stop gaming the number, because there is no number to game.
Be honest about what you can't yet claim
Here is the part vendor dashboards will never show you: most of this is young, and the honest answer to "did it improve outcomes" is often "I don't have the before/after data yet." Say that in print. An evaluator that scores a plan as strong has told you the plan reads well, which is usually but not provably the same as executing well. Flag the unvalidated loops as unvalidated. Leaders who admit what their data doesn't support are the ones worth believing when it does.
Take what helps
Measuring AI adoption well is mostly a matter of measuring the right thing: the quality of an artifact and the trajectory of a person, not the height of a usage bar. Get the signal from the work, send the data to the individual first, alarm only on wheel-spinning, and never build the scoreboard. Your team and your tools will differ. Take what helps, skip what doesn't.
The measuring-adoption playbook is this same approach with the numbers and the mistakes from one real team.