Join me exploring Microsoft Copilot through each letter of the alphabet. This week, K is for KPIs.

Specifically: how do you actually measure whether Copilot is working? And what does “working” even mean?

My situation vs. yours

I’m very aware my work life might look different to your work life. 

I work in a company of eight people. We value tech, we move fast, and Copilot/AI licences are handed out with a “use it or lose it” energy rather than a “justify this to the board” conversation.

We even have a Teams group chat called How We AI where we share what we’ve been doing with AI tools; wins, disasters, weird experiments, all of it. I think (hope) it’s helped us to show each other what’s possible and drive up the ‘use-AI’ muscle memory.

If you work in a big company, you probably have a different experience. 

For a large organisation, Copilot licences represent a serious chunk of budget. Per-person, per-month, multiplied across hundreds or thousands of employees. Adds up to some mind boggling numbers no doubt. How do you prove it’s worth it? That’s where KPIs come in.

The dashboard answer

Microsoft gives some useful tools here. The Copilot Dashboard in Viva Insights shows you who’s using Copilot, what they’re using it for, how often, and who hasn’t touched it since the rollout email. You can pull estimated productivity metrics; time saved on email, meeting summaries generated, that kind of thing. Copilot Business Impact report | Microsoft Learn 

For the “justify this investment” conversation, these numbers are a starting point. They’re real, they’re trackable, and they give you something to put in a slide.

I suspect they only tell half the story. Possibly less.

Microsoft MVP Kevin McDonnell agrees. (A big thanks by the way to Kevin for kindly contributing his thoughts to this post. I asked if he’d share his expertise on measuring Copilot value, and he very generously obliged. Go give him a follow on LinkedIn.)

“Measuring the value of Copilot is key to justifying what is a not insignificant investment in Microsoft 365 Copilot but it is also important to give a wider view of value than just Return on Investment. Many focus on numbers from the Copilot Dashboard such as time saved and amount that Copilot is used in different areas. However, the impact to how employees feel with Copilot usage, the improvement they feel to their work, the increase in quality of what they can deliver and how well they feel that they are able to use it – these all play into the overall value. Combining the quantitative with the qualitative is what provides the true value to an organisation but is the extra complexity to deliver that many organisations don’t focus on.”

Kevin McDonnell, Microsoft MVP

Good to know I’m not making this gut feel up. 

The question nobody wants to answer

Here’s the uncomfortable one: if people in your organisation aren’t using Copilot, what does that actually mean?

Is it a Copilot problem? A people problem? Or an organisation problem? Did anyone actually show people what’s in it for them, specifically, in their role, on their actual Tuesday afternoon?

Low usage on a dashboard looks identical whether the tool can’t solve someone’s niche use case, or whether they tried it once, it didn’t quite work, and nobody followed up. Those are very different situations requiring very different responses. The number doesn’t tell you which one you’re looking at.

The hidden KPIs; good and bad

If we’re going beyond the dashboard, here’s what I’d actually be looking for:

The “I didn’t feel stupid” moment

 In J is for Joy I mentioned using Copilot to decode dense contract language so I could have an actual opinion in a meeting without quietly Googling “what does notwithstanding mean.” Nobody measured that. But removing the friction of “I don’t know and I don’t want to ask” is real value, especially for people newer to a role or less confident in a particular area.

The “I actually read it” moment 

How often do people receive a long document, report, or email chain and quietly file it under “will get to that”? When Copilot summarises it into five bullet points, it’s much easier to get people to engage with the content. Comprehension goes up. That’s not measurable, but it’s real.

The note-taker is finally free 

If you’ve read F is for Facilitator, you’ll know I used to be the designated meeting scribe; furiously typing while trying to also add value. Since Facilitator is taking the capturing load, I ask more questions, I follow threads I’d have dropped, and I’m actually in the room. How do you measure the quality of someone’s thinking in a meeting? You don’t. But it changed.

The lightbulb in someone else 

There is a very specific joy in saying “have you tried asking Copilot to help with that?” and watching someone go from sceptical to “oh. oh.” That mindset shift is a KPI. It just doesn’t have a column anywhere.

The meeting that didn’t need to happen 

If someone can ask Copilot for a status update and get a decent answer from existing documents and chats, that’s one fewer “quick catch-up” in everyone’s diary. You can’t count meetings that didn’t happen. But they didn’t happen.

The thing someone did themselves instead of outsourcing 

When someone creates a half-decent graphic, drafts a first pass at a policy document, or builds a basic data summary without needing to ask a specialist colleague, that’s capacity freed up on both sides. Skills gaps quietly shrinking.

My personal favourite Copilot metric is what I call the cup of tea moment: the point where you set it working on something, go and make a brew, and come back to a finished first draft. I’ve lived this. It’s audacious and I love it. No ‘cups of tea made’ will show up on anyone’s dashboard though. 

There are hidden costs as well as gains

Here’s the one that doesn’t make it into the “celebrate Copilot” slide decks: the time spent correcting AI mistakes.

Checking outputs. Fact-checking statistics. Rewriting the bits that sound like a corporate policy memo rather than a human being. Re-doing the PowerPoint that confidently presented 45 fictional new hires.

That time is real. It doesn’t show up as a cost anywhere, but it absolutely is one. Honest measurement of Copilot’s value has to include it, otherwise you’re only counting the wins.

So how do you actually measure it?

Kevin points to Microsoft’s Viva tools (Engage, Pulse, and Insights) as ways to capture the qualitative side alongside the numbers. Regular pulse surveys. Employee sentiment. Honest questions like “did this save you time?” and “did it make your work better?” rather than just “did you open it?”

The organisations that get the most from Copilot, I suspect, treat adoption as an ongoing conversation rather than a rollout event. Not “we’ve deployed it, use it” but “here’s what it can do for you specifically; let’s find out together.” And then actually following up when people don’t.

For me in our company of eight, with our scrappy little How We AI chat ticking away in the background, success looks like one less frantic inbox search, one meeting where I was actually present, one contract clause that made sense without a law degree. Small things. Stacked up week after week, though? That’s the value.

TLDR: The Copilot Dashboard will tell you who’s using it and roughly how much. That’s worth knowing when you have to justify the spend. But the real picture needs the numbers and the humans. Measure the cup of tea moments alongside the correction time. And if people aren’t using it, ask why before assuming you know.

What does Copilot success look like in your organisation? And has anyone found a brilliant way to measure the unmeasurable? Drop a comment, I’d love to know and so would others I’m sure.

Useful links:

• Copilot Business Impact Report: Copilot Business Impact report | Microsoft Learn  
• Kevin McDonnell on LinkedIn: Kevin McDonnell | LinkedIn