Playbook

Measuring AI Adoption on an Engineering Team, Without a Leaderboard

I lead a team of ten engineers. I told them to use AI. Then I discovered I had no way to see whether anyone was, whether it was any good, or who needed help. And the vendor's own analytics hid my team behind hashed IDs.

Mary, Director of Engineering, 20 YOE

The situation

A year ago, one of the other directors on our floor introduced Devin, our first real AI coding tool. I was hooked quickly, but I saw its limits just as fast, so I never went as far as declaring us an "AI-first" team. I told people to take a look, was upfront about what I didn't trust yet, and left it there for a while.

That held until somewhere between Q3 2025 and early this year, when I started working with Claude and watched the models get noticeably stronger. I never bought the hype or the money-grab pitches. I poked at it, pushed it, tried to find where it broke, and watched it keep getting more capable. That's when I started pushing my team harder: love it or hate it, I told them, this isn't going anywhere, and it's already changing the shape of the job. Make friends with it.

Like most engineering directors right now, I was expected to drive AI adoption on my team, and I did the obvious thing first: I told everyone to use it.

"Everyone should use AI" fails as a directive for a boring reason: it isn't falsifiable. Nobody can be doing it wrong, so nobody can be helped. In practice a mandate like that decays into one of two failure modes. Either you get box-checking (the tool gets opened, nothing about the work changes) or you get silent non-adoption, where the engineers who most need the leverage quietly keep working the old way, because nobody can see the difference anyway. Either way, I'd find out the truth the slow way: a sprint slips, and only then do I go looking for why.

The constraint

There was a second, dumber problem stacked on top. Even when I wanted to see who was using what, I couldn't. Our AI tooling's usage analytics showed per-user activity, but behind salted-hash identifiers, with no admin API on our plan tier to resolve them. I could see that someone was our heaviest user and someone had barely logged in, with no way to know who was who.

You cannot coach a hash.

What I did

Two moves: pick one habit that leaves evidence, and build the visibility to see it.

The habit. Instead of "use AI more," I set one specific expectation: before writing code, use Claude's plan mode to draft a plan.md (approach, edge cases, what could go wrong) and attach it to the ticket. The attached plan does double duty. It's proof the habit is happening (the file either exists on the ticket or it doesn't), and it's an artifact I can actually inspect: the engineer's real thinking, on the page.

The evaluation loop. Periodically I pull in every engineer's plan files and have AI evaluate each one individually. Not "is there a plan attached" (that's the box-checking trap) but how strong is the thinking: does it anticipate the edge cases, is the approach sound, what's missing? Each plan comes back with specific, constructive feedback, and I hand every engineer their own feedback directly. The same pass that shows me who's struggling gives them exactly what they need to improve without me. That's the difference between "my manager is grading me" and "I have a loop I can run myself."

That started as a command I ran. It is now a page I open once a sprint.

Two things in that header came out of running it for real, and neither was in my original ask. Both are about what you measure adoption against.

• Adoption is measured against eligible tickets, not all tickets. "34 of 42" is the honest denominator. Plenty of tickets do not want a plan: a copy change, a config bump, a one-line revert. Measuring those as misses would have taught everyone that the number is nonsense, and a number people believe is nonsense is a number they stop reading.
• "Exempt" is a labeled state, not a silent gap. Anything deliberately excluded gets tagged and counted in its own tile. The difference between "nine tickets we decided don't need a plan" and "nine tickets missing a plan" is the whole difference between a measurement and an accusation, and if you don't build the first one, everybody assumes the second.

Per ticket it shows the plan file (or a missing warning), a quality score, and the actual note. When a ticket has more than one plan attached, it scores each and shows an averaged overall row rather than picking one. The notes are the part I check for rot: the moment they turn generic, the loop is dead. A good one reads like the one in that figure, naming the specific thing the plan skipped and what to add instead. That is the rubric working. "Consider adding more detail" would mean it had stopped.

The visibility layer. I'd built a hash-to-human join like this once already, while chasing down which seats were driving our AI spend. Reusing it for coaching visibility took genuinely annoying work: joining two of the vendor's data exports on exactly-matching cost values (the one field both sides happened to share) to map hashed IDs back to real people. Fragile, manual, slightly absurd, and worth it. That mapping let me put per-person usage, seat assignments, and spend on the dashboard I already run for the team.

Two design choices in that view matter more than the plumbing:

• The dashboard says plainly that the vendor's "spend" figure is an estimated value of usage, not billed dollars. I won't present a derived number as a real one.
• The only usage signal with an alert on it is wheel-spinning: lots of sessions with little or nothing accepted. Not "too little usage," not "too much spend." High activity with nothing accepted means a person is spinning their wheels with the tool. (That one grew into its own page, below, and growing it taught me more than the principle did.) That's the one usage number that's a coaching signal rather than a policing one. Spend anomalies are a budget conversation. Wheel-spinning is a mentoring conversation, and mentoring is the job.

One explicit non-goal, and I want to be blunt about this one: no leaderboard. The moment usage numbers become a scoreboard, people stop optimizing for the work and start optimizing for the number. On an AI usage board, that number-chasing has a name, token maxing: running more prompts, longer sessions, more tool calls, not because the work needs it but because the dashboard is watching. It's the direct, predictable result of a "use AI more" mandate with no artifact behind it: you told people to move a number, so they move the number. A team that's token-maxing looks busy and looks compliant, and is neither thinking harder nor shipping better. That's the trap I built this whole approach to avoid: coach the plan, never the count.

I want to be precise about what "no leaderboard" rules out, because the version of it I actually run does name people. The plan tracker puts one developer's name under "needs coaching" right in the summary. That is a flag, and a flag is a list of the people something should be done about. A leaderboard is a ranking of everyone, including the twenty-nine people nothing needs to be done about. The rule isn't "never name anyone," it's never rank the unflagged. If I couldn't name the person who needs help, the tool would have no reason to exist. If I ranked the rest of them, it would have a reason to be deleted.

The one alert, and how it stays a conversation

The wheel-spinning signal got its own page, and building it forced three decisions I had not thought hard enough about when it was just a principle in my head.

One threshold was not enough. The clean signal is many sessions and zero accepted output: that person is stuck, full stop. But there is a softer pattern, lots of sessions with a low accepted-lines-per-session ratio, that is worth a look and is nowhere near as conclusive. Collapsing both into one alert would have meant either missing the soft cases or crying wolf on them. So there are two tiers with different names, and the softer one is explicitly the weaker signal. In the run in that figure the strong tier is empty, which is a result, not a blank.

The disclaimer belongs in the product, not in my head. Low accepted output can mean somebody is stuck. It can also mean careful review, or a week of hard debugging where the right amount of generated code is very little. So the page says that, right under the list, every time. I wrote it for me on the day I would be tired and reading fast, because the failure mode here isn't a bad metric, it's a manager acting on a soft signal as though it were a verdict.

Ambiguity gets its own section, never the flag list. Some people have no seat on record. Some are in scope but have no usage rows in the window. Neither of those is evidence of anything, and both are extremely easy to quietly fold into "low usage." They're listed separately, under a heading that says they are not flags. A measurement tool that hides its own gaps is just a confident tool.

One more piece of honesty I'd steal for any metric with a window: the thresholds scale to the data that exists. When I first ran it there was no fourteen-day baseline yet, so it used a shorter month-to-date window and said so on the page, and it switches to the real rolling window on its own once there is enough history. A number computed over a window you can't see is a number you can't argue with, which sounds like a feature and is not.

And it does not message anyone. The page says that too. Running it produces a list for me to act on as a human, not a bot that tells an engineer their metrics look bad.

What's happened so far

Honest results, with the caveats attached:

• Adoption is now observable instead of anecdotal. Whether the habit is happening is a yes/no per ticket, and per sprint it's the adoption tile in the figure above. Before, it was asking in 1:1s and hoping.
• I spot struggling engineers earlier. Weak thinking shows up in plan quality and in the stuck-signal weeks before it would surface as a missed sprint. Early intervention is the concrete win I can already point to.
• Engineers get feedback they can act on without me. Per plan, specific, theirs.
• The pushback turned out to be the point. When I'd hand someone the AI's take on their plan, they'd often push back: explain why the flagged issue wasn't actually a problem, and why. It could read as excuse-making. It isn't. It's an engineer reasoning out loud about their own plan, about the AI's read on that plan, and about how the work actually went once it shipped. That's three layers of thinking I wasn't getting out of "just use AI more."
• What I can't claim yet: that the coaching measurably improves outcomes. I don't have before/after data showing that engineers who received plan feedback write better plans, or deliver more smoothly. The loop is young. The trend I care about most (is the same engineer's tenth plan stronger than their first?) is the number I don't have yet.
• A second honest flag: the evaluator itself is unvalidated. The AI's judgment of "a strong plan" hasn't been checked against what happened when the plan was executed. Until I close that loop, a plan that scores well is a plan that reads well. Usually the same thing, not provably the same thing.
• What I still want to dig into: real numbers on how much the team uses AI, when, which models, and how delivery quality tracks against token spend. The cost-driven mapping I described above answers "who's spending what," not "who's actually getting better output for it"; that's the gap I want to close next.

What I'd do differently

• Baseline from day one. I started evaluating plans after the habit was already running. If I'd scored the first batch before any feedback went out, I'd have the before/after trend I'm now missing.
• Pick the artifact-leaving habit first, not after the slogan fails. I spent real time on "embrace AI" without actually defining what that meant, before switching to "attach a plan to the ticket." The directive phase produced nothing I could see or coach. Skip it.
• Turn the mapping into a tool, not a one-off. My first instinct was to escalate the analytics gap to the vendor. In hindsight that was never going anywhere: the opacity costs Anthropic nothing, so there's no reason for them to make identity resolution easier. The better move is writing the hash-to-human join as a small reusable script instead of a one-time hack, so it survives the next export-format change instead of breaking silently.
• Watch for rubric-gaming sooner. Once engineers know plans are evaluated, plans can be written for the evaluator instead of for the work. I haven't seen it yet, but the incentive is there. The mitigation I trust most: keep the feedback genuinely useful. A loop people want to run isn't one they game.

Two limits I've accepted rather than solved: one habit is not a practice (reviewing with AI, test design, and debugging would each need their own inspectable artifact), and the coaching ceiling is me. The evaluation pass would scale to fifty engineers, the conversations wouldn't.

Take what helps

Most AI-adoption failures I've watched up close weren't tooling failures. They were visibility failures. Leadership declared a direction, couldn't observe whether anyone was moving, and quietly stopped steering.

If you take one thing: pick a single AI habit that leaves an artifact, and coach the artifact. Usage data tells you where to look. The artifact tells you what to say. The dashboards are just substrate. The product is each engineer getting better at knowing what to hand the machine and where a human stays in the loop. If the dashboards became the point, I'd have failed.

Your team and your tooling will differ. Take what helps, skip what doesn't.