How to Measure AI Impact (Not Just AI Adoption)

Most organisations track AI adoption, not AI impact. Learn how to measure what AI achieves against your purpose, with two worked examples.

Every leadership team now feels the pressure to adopt AI. This pressure comes from boards, ministers, competitors and vendors, and the fear of being left behind in a rapidly shifting world.

But there’s a second pressure nipping at the heels of the first: to prove that the investment in AI is paying off. And as we already know from our experiences trying to measure anything complex or intangible, it’s where most organisations get stuck, resort to activity metrics, or simply give up.

It’s tempting to conclude that AI is so different that measuring its impact needs a whole new discipline. We don’t think it does. AI is a subject for performance measurement, in the same way that customer experience, environmental impact, employee engagement, or DEI are subjects for performance measurement. Researchers don’t reinvent the scientific method every time they study something new, and measurement shouldn’t be reinvented either.

What AI does change is where the implementation of measurement gets difficult. AI’s adoption is more systemic, its operation is more invisible, and its consequences are more pervasive. Its effects can arrive late, its technology shifts quickly under our feet, and the data to understand it is elusive. Yet much of the advice on offer jumps straight to a list of easy-to-track metrics that ignore all these realities.

This article follows the same thinking process we use to build any performance measurement framework, and points out where AI needs extra care at each step. To keep it practical, we’ll follow two fictitious organisations through it: a state main roads department and an animal rescue organisation. Both are illustrative, so even though our approach has been used in similar organisations, the examples in this article are not client cases.

Why it’s worth getting AI measurement right

Two predominant reasons exist for why we have to measure AI’s impact better than we currently are.

Reason 1: Measuring AI adoption is the fast road to the wrong destination

AI is being adopted faster than its impact is being evaluated. That’s because impact is harder to measure. But measuring only adoption creates a knowledge gap, where our ongoing adoption is not informed or directed by what works and what doesn’t. And what works can be uniquely different in each of our organisations.

Across the OECD, 35 of 36 countries now use AI in at least one area of government, yet only 10 of 36 (28%) report any financial or non-financial impact measurement of AI use cases, according to the OECD Digital Government Outlook 2026.

The private sector looks much the same. McKinsey reports that nearly eight in ten organisations use generative AI in at least one business function, yet 60% of survey respondents have not seen enterprise-wide EBIT impact from their AI programs. In BCG’s latest CEO survey, many CEOs recognised the need to link AI initiatives to the P&L, but only 14% clearly defined the P&L impact for all of them.

Without getting a handle on measuring the real impacts of AI, the costs of adoption can overwhelm the benefits long before anyone realises, leaving it too late to direct its course.

Reason 2: AI creates unintended impacts that aren’t always visible

Measurement can certainly tell us whether AI is having the desired impact. But it also helps us see where it may be creating unintended impacts, desired or not. When our only measures are speed, volume or usage, we’re allowing the side effects to grow, unwatched.

The Georgetown Beeck Center puts the government version plainly. An AI system that catches more fraud but wrongly cuts off eligible people from benefits has failed, even if the numbers are green on the dashboard. Success, it argues, is whether people get “faster, fairer, and more reliable access to the services they are eligible for” — not how many AI use cases an agency can list. There were 3,611 use cases across 56 US federal agencies in the 2025 inventory, up from 1,757 the year before. That’s very easy to count. But it says nothing at all about whether people’s experience of government is getting better. Or worse.

The Ada Lovelace Institute found the same gap in UK public sector productivity claims from AI adoption. Most studies measure first-order effects such as time savings or cost reductions, not outcomes like service quality, equity or worker wellbeing. And in Harvard Business Review, Ben Waber and Nathanael Fast caution that productivity gains seen at the task level can’t be assumed to hold for whole-of-firm performance, and that the long-term effects on employees and internal processes may be negative.

The price of not knowing can be far greater than wasted spend — we must add to it the hidden costs of misdirected effort, rework, and delayed action. In some cases, the costs also burden others outside the organisation itself.

None of this is an argument against AI.

Every change brings its complexities. But this is an argument for deciding what we should look at before the technology is locked into how our organisations do the work they exist to do, so the results that matter to people drive it from the start.

Executives need confidence they can defend to a board or a minister or an auditor when someone asks what AI has actually delivered. Planning and performance professionals need a framework that will still stand up in twelve months, when the first enthusiastic targets meet real data or changing priorities. Both are served by the same thing: a measurement approach that is deliberate about what the organisation is trying to achieve before it chooses a single AI metric.

But before diving head-first into an AI impact measurement initiative, there are a few considerations that will help us build a strong foundation for it.

Why AI is harder to measure than it looks

Measuring AI’s impact has the usual difficulties of measuring any kind of change, plus three that are more particular to AI.

The usual difficulties: the value is indirect and often intangible

The first-order effects of AI are easy for us to see: less time per task and more output per person. But many of its benefits are second-order effects. BCG gives the example of faster, more accurate decisions: they improve performance, but how much, and by what route, is hard to predict in advance. Effects like these are less tangible and harder to trace back to the tool. That’s familiar territory for anyone who has tried to measure the impact of a new technology or change on organisational performance.

The answer isn’t specific to the type of change we’re adopting. The answer — always — is to be clear about the result we’re seeking to improve before we choose how to measure that change.

Difficulty 1: the ground keeps moving

The technology changes while we’re still measuring it. BCG notes that plans have to change as the technology evolves: a two- to three-year transformation plan will need adjusting several times, in response to developments no-one can predict today. An impact we desired at launch may not be the impact we discover is more important six months later. And even when it still is, there’s often a dip, the J-curve, before AI breaks even or actually lifts performance.

Measuring only what stays constant isn’t the answer. Building the muscle to quickly measure what matters is.

Difficulty 2: the data sources are unreliable

The Ada Lovelace Institute points out that a growing number of AI productivity studies use large language models to judge which tasks large language models can automate. That’s circular logic. Self-reports are no more objective. The same briefing describes a study in which experienced software developers estimated that AI had made them 20% faster. When their actual output was measured directly, they were 20% slower! Headline figures can be just as rubbery: one UK government estimate of annual AI savings was £45 billion, and a later version of the same estimate was £23.6 billion.

The solution isn’t to skip measurement because of imperfect data. If we design evidence-based measures for the AI impacts that matter most, we have a better chance defining the data that can do the job.

Difficulty 3: most advice gives you a list, not a model

Much of the AI measurement advice we reviewed offers a list of metrics: adoption rates, hours saved, cost per interaction, satisfaction scores. Some of it might be useful, but because a list has no logic holding it together into a single theory of change, there’s no way to test it or tailor it. Everything looks equally necessary, which is why AI measurement can feel overwhelming and broader than it needs to be. Even McKinsey’s more structured five-layer framework, which runs from technical performance through adoption and operations to financial results, describes layers every organisation can relate to, but not the cause-and-effect logic that lets an organisation test and adapt it to its own situation.

No single framework exists that an organisation can adopt to measure AI impact. What we need instead is a way to map our theory of change for how AI will lead to the gains we expect, and helps us see where we left gaps or made logic leaps.

If we don’t address the difficulties, mistakes are inevitable

These difficulties are behind the common AI measurement mistakes we all keep seeing and reading about. Most are all as old as measurement itself:

• Counting activity. Numbers of AI users, prompts or use cases show that AI is being used, not what it achieved. Tracking AI adoption reveals little about whether AI is having a worthwhile impact on the organisation’s achievement of its purpose.
• Stopping at outputs. Hours saved sounds like an impact, but it’s an output of AI adoption. We only get a positive impact if someone does something even more worthwhile with those saved hours.
• Looking only at the upside. Additional costs, trade-offs and other unintended consequences aren’t uncovered and measured, and can’t weigh in on decisions that seek to optimise the organisation’s performance as a whole.

• Trusting evidence that’s easy to get. Self-reported savings, vendor-supplied numbers and AI grading AI are all convenient, but none of them is unbiased, reliable and direct evidence of impact.
• Setting linear targets. In a field that keeps changing and follows J-curve improvement ramps, traditional straight-line targets can lead to the wrong responses and unnecessary knee-jerk reactions when real-world results miss them.

None of this means AI can’t be measured well. It means the framework has to be built deliberately, not by picking metrics off a shelf. A list assumes every organisation wants the same things from AI, and none does. The place to start is purpose.

It starts with purpose, not with AI

While all our organisations can learn from best practices in adopting AI, the impact AI should have is different for each, because it depends on why each organisation exists and what it needs to achieve.

“It depends” is a truism but we can’t ignore it here, because AI measurement advice so often starts at the wrong end. It starts with the technology (usage, hours saved, accuracy) and works its way up towards the organisation’s purpose. We need to start with the organisation’s purpose and work down to the technology. BCG makes a similar point about the CEO’s role: the foundational questions are how AI supports the company’s strategy, what it should deliver, and how success will be measured. And the Ada Lovelace Institute calls for a clearer statement of what public sector AI should achieve beyond “productivity” for productivity’s sake.

AI doesn’t change what our organisations exist to achieve. It only changes some of the ways we might achieve it. So two questions come first, before we think about AI at all:

1. Why do we exist? Who do we serve, and what difference are we here to make?
2. What results have we decided to improve right now, to fulfil that purpose better?

Only then can we ask the AI question: where could AI help us achieve those results, and where could it get in the way?

Two organisations, two different answers

To see how this plays out, meet our two fictitious organisations.

	Riverbend Main Roads Department	Harbour Lane Animal Rescue
Purpose	People and freight move safely and reliably across the state’s road network, including the regional communities that depend on it.	Rescued and surrendered animals are rehomed healthy and new owners continue excellent care for their pet.
Current strategy	The population is growing faster than road capacity can be built, so the department is getting more out of the existing network by making smarter decisions about where limited maintenance and upgrade funds go.	Demand for care is outgrowing what the clinical team can deliver. The clinic aims to treat more animals without compromising clinical quality or burning out its vets and nurses.
AI initiative	AI-supported prioritisation of maintenance and investment. A model ranks road segments using traffic volumes, crash history, sensor data and benefit-cost estimates, and planners review its recommendations.	AI-assisted clinical note-taking. Vets and nurses spend less time on documentation, and the drafted notes are reviewed before going into each animal’s record.

Notice that neither organisation started with “let’s measure AI”. Each started with a purpose and a strategy, and the AI initiative is one possible contributor to them. The same tool would be judged against entirely different results in each. A faster clinical note is only good news at Harbour Lane if more animals get the care they need and the records don’t get worse. A faster prioritisation is only good news at Riverbend if the roads that regional communities depend on aren’t the ones falling off the list.

These two examples will help us explore how we can choose the AI results that matter to our own organisations.

And we’re not starting from a blank sheet

Purpose and strategy decide most of what we need to measure, but a few AI-specific results matter to nearly every organisation: how quickly teams adapt as AI keeps changing, the safety and accuracy of the data AI uses and produces, and how widely and well people use the tools. These are enabling results. They don’t show that AI is working, but they influence whether — and how well — it can. The results that show what AI achieves are specific to each organisation, and harder to pin down.

Most AI impacts are organisation specific

Again, purpose and strategy decide most of what we need to measure about AI. If we haven’t done the thinking to produce a map of the strategic results that matter to the organisation, we have no foundation to build our AI theory of change — how the effects of AI adoption flow through the map and which results they touch, either positively or negatively. We use a Results Map to make this easier to see.

A Results Map is a one-page picture of the results an organisation wants to achieve, and how they affect each other. Each result is a statement of what success looks like. Arrows show cause and effect: achieving this result helps cause that one. And where improving one result tends to damage another, we draw a conflict, so we can deal with the tension deliberately instead of discovering it when it’s too late. It helps us trace the chain of effects of AI adoption throughout the entire organisation.

A real example of an AI impact chain

Here’s a real example of the kind of chain this map makes visible. In a study of a generative AI assistant used by 5,179 customer support agents, Brynjolfsson, Li and Raymond found that productivity rose by 14% on average. But the same tool also improved customer sentiment, reduced requests to escalate to a manager, and reduced attrition among newer agents. Productivity, customer experience and retention were linked results, and measuring only the first would have missed the rest.

The study also shows why counting adoption easily misleads us. Customer support agents followed the tool’s suggestions only about 35 to 38% of the time on average, and the most active users gained far more than the least active. A measure of “access rolled out to all agents” would have looked like success while saying nothing about the use or usefulness of the tool in practice. In that case the impact chain was measured because the researchers happened to have the data. Most of us won’t be so lucky. We have to map it.

How to map the AI impact chain

We start with goals we already have in our strategic direction, and then build the cause-effect flows from our AI initiatives through those goals. The goals that the AI initiatives touch are our AI impacts.

We’ll see how a Results Map makes this impact chain visible and straightforward. But how easy it is rests on another, less obvious requirement: clarity of our goals. If our goals are not written clearly enough, it will be harder to see which ones our AI initiatives touch, and harder to measure them to gauge AI’s impact.

Making our AI impacts clear, specific and measurable

Intangibly worded goals are a common problem in general and not specific to AI impacts. We won’t dive into it here, because we already have a lot written about solving this problem, such as dealing with action-oriented, weasel-worded, and multi-focus impacts. This work is important to do before mapping an AI impact chain.

And this work has already been done in our two cases: Riverbend and Harbour Lane. This is why their Results Maps, each highlighting an AI impact chain, will be easy to follow.

Riverbend: testing the logic of the AI-supported decision systems

Here is the slice of Riverbend’s Results Map that matters for its AI-supported maintenance prioritisation and AI-assisted traffic signalling initiatives. It’s deliberately small. Only results that serve the chosen strategy appear.

First, read it from the outside in. Two AI initiatives (the red boxes) are to support maintenance prioritisation and to assist traffic signalling. Each initiative will have its own AI adoption metrics, but the important effects only become visible when linked to the Results Map. AI-supported maintenance prioritisation should, first and foremost, improve the timeliness of road investment prioritisation, at the same time as putting more road investment where it creates the most value. If it can make prioritisation more timely, then road condition will cause fewer accidents and produce fewer bottlenecks. These improvements, in turn, will help travel times to be more predictable and traffic incidents less frequent and less severe. And as long as it also makes sure investment is going where it creates the most value, the unintended consequence of any community being isolated can be avoided, and communities can stay connected.

Reading a Results Map from the outside in tells the theory of change story, from cause to effect. But that’s not how they’re built. A Results Map, or theory of change, is built from the inside out.

Harbour Lane: testing the logic of AI-assisted clinical note-taking

And here is the slice of Harbour Lane Animal Rescue’s Results Map for its AI note-taking initiative:

There are three chains here, for how AI-assisted clinical note-taking impacts Harbour Lane’s performance. The first is the one most AI measurement assumes: less time on notes should free up time for animals, and so more animals should get care. Saved time only becomes more animals treated if someone actually redirects it, and measurement would help us test that link rather than assume it. The second chain holds a conflict — faster notes pull against accurate notes, because a plausible-sounding mistake in a clinical record is easy to miss, and a vet checking each drafted note before it is filed is how that conflict should be managed. The third chain is the human one, and it holds a second conflict: if all the freed-up time goes into treating more animals, vets and nurses may be as stretched as before. Whether they want to stay depends on how the saved time is shared between more animals and a manageable workload.

The maps make hidden AI impacts suddenly visible

Tracing the impact of AI initiatives through a Results Map is the easy way to see all the important impacts it should and could have on what matters in our organisation:

• Productivity leaves its silo. Time saved is one early link in a chain that feeds up to purpose, so we can see what has to be true for it to matter.
• The human results sit on the same page as productivity. The conflicts are visible before we’ve chosen a single measure.
• Every arrow is a claim we can test. Does faster prioritisation really send funds where they create the most public value? Only if the model values regional roads and community connectedness properly.
• The map stays small. Only results that serve the chosen strategy appear, which avoids the long, flat lists that make AI measurement feel overwhelming.

With the results clear, choosing measures is the next step. And clear results make that step much easier than anyone might expect.

With the results clear, deciding what to measure gets simpler

For each result on the map that matters, we ask what we would see more of, or less of, if it were achieved, and we quantify that. We have solid techniques in PuMP to guide this, like the four-ingredient recipe for writing quantitative measures. These techniques flow easily when the results we’re measuring are written measurably.

The map also tells us which measures belong together

Riverbend needs a measure of how timely its investment prioritisation is, but never on its own. It needs one alongside it for whether investment is going where it creates the most value, regional communities included.

Harbour Lane needs a measure of how quickly clinical records are completed alongside one for whether those records are accurate and complete. Measuring both sides of a conflict is how we stop one result improving at the expense of the other without anyone noticing.

Measuring AI ROI is a natural extension

Once we can see the impact AI is having, return on investment is a short step further. ROI compares the dollar value of AI’s gains with the total cost of getting them: the value of the gains minus the total cost, divided by the total cost.

We already have the first ingredient, because the measures on our Results Map show the gains in the results that matter. The second is translating those gains into dollars, wherever a defensible conversion exists, such as costs avoided or revenue earned or value created for customers or communities. Hours saved only become dollars if the time is redeployed or the cost is genuinely removed. The third is the full cost of adoption: licences and usage fees, implementation, training, the ongoing time spent checking and correcting AI outputs, and the other side of those conflict relationships.

ROI is not the most straightforward thing to measure. Not every result converts to dollars, and fairness, trust and community connectedness are at least as important as financial impact. And that AI J-curve, the dip before AI gains arrive, needs to be taken into account.

Whether we take that extra step to measure AI ROI, or stop at AI impact measures, what we can learn is only as good as the data behind it.

Data we can trust is the bigger challenge

Choosing a measure is an important and deliberate step. Finding data that tells the truth is another matter, and this is where AI invites shortcuts. Usage logs, vendor dashboards, self-reported savings and AI grading AI are all easy to get, and none of them is direct evidence of impact.

Better evidence comes from the work itself: records, outcomes and independent checks. For Harbour Lane’s notes, that could be a regular sample of AI-drafted notes, checked by someone who was in the consult against a simple list of what an accurate and complete note includes. That matters more for AI, because the same prompt can produce different answers. The OECD notes that governments are starting to audit samples of AI outputs against predefined criteria for just this reason. The time measures need the same care. Minutes spent documenting per animal can come from records-system timestamps, and the share of vet time spent directly with animals from systematic logging, not from vets’ own estimates, because as we saw earlier, people’s own judgement of AI’s time savings can be wrong.

For Riverbend, the evidence that investment goes where it creates the most value comes from two places. The department’s records show how many roads with reduced speeds or closures are still waiting for funding. And a small, regular review by senior planners of unfunded road segments (regional roads included) shows how many should have been prioritised, which tells us early whether the AI is steering funds away from roads that matter. The records tell us where decisions have led. The review tells us whether the AI-assisted maintenance prioritisation is making any wrong calls.

Some of the evidence for our AI impact measures won’t exist yet. That’s normal. Designing how to collect it is part of the work, and it’s best started before the AI goes live, so we have a baseline to compare against.

The rest of the process: pace it, and keep people in it

Collecting the data, displaying it, spotting the signals in it and deciding what to do about them works the same way for AI as for anything else. Where AI needs extra care is in the pace we expect and in who we involve.

Set targets that expect the J-curve

AI’s benefits can take longer to arrive than the investment in them, and results can lag or even dip while people learn new ways of working and processes are redesigned. Economists Brynjolfsson, Rock and Syverson call this the productivity J-curve. A target that demands straight-line improvement from day one makes an expected dip look like failure, and tempts us to either abandon something promising or push people harder. Instead, set targets as a trajectory that allows for the dip and the climb. Review results in short cycles, and use valid signals to tell real change from normal variation before reacting. And because the technology keeps shifting, revisit the Results Map and the measures regularly.

Don’t do it without the people who live with the results

Leaders and the people doing the work see AI differently. In McKinsey’s 2026 survey, 47% of mid-level managers and individual contributors reported AI-related strains, compared with 31% of executives and senior managers. If we only ask leaders, we’ll miss much of what’s happening.

So involve the people the AI touches when we map the results and decide on measures, when we judge the evidence and when we interpret the signals. Harbour Lane’s vets and nurses will sense long before any dashboard whether the notes are accurate and whether the saved time is real. Riverbend’s planners will recognise which regional roads the model scores strangely. Their knowledge is the early warning system, and a guide for the right measures and the best data sources.

It’s achievable, and we’re not starting from a blank page

That’s the whole path, and it’s the same one we’d take for any result that matters:

• Start with purpose and strategy, not the technology.
• Map the results AI could affect, including where it could help one result at the expense of another.
• Choose measures for those results, keeping both sides of every conflict in view.
• Find evidence we can trust, not just evidence that’s easy to get.
• Set targets that expect a J-curve, and read the results for signals before reacting.
• Keep the people the AI touches involved all the way through.

AI is new terrain, but it isn’t a new measurement discipline. The method we use to measure other hard-to-measure things works here too, because it starts where AI measurement so often doesn’t: with what our organisation exists to achieve. Riverbend and Harbour Lane aren’t special cases. Any organisation can start small, with one AI initiative and one map, and build from there.