KPI’s What You Measure is What You Get

The goal of a KPI, a Key Performance Indicator, is to measure an activity or outcome that contributes to the success of an organization.

Often, a KPI is a leading indicator. It gives us a signal about what is likely to happen before the numbers ever appear on the income statement or balance sheet.

For example, student enrollment or student inquiries can provide an indication of future tuition revenue. In a manufacturing company, the order book may be a critical indicator. In one company I worked with, we wanted to maintain 90 days of orders on the books.

Other organizations might focus on billable hours, customer retention, utilization, or warranty claims. In the case of warranty claims, the impact is much more direct: eventually, those claims hit the bottom line.

But there is a problem with KPIs. People respond to what you measure. For outbound sales, for example, a KPI might be the number of calls made, meetings set, or proposals sent. These can be useful measures of sales activity. But if salespeople are rewarded primarily for those activities rather than actual sales, the organization has created an opportunity to game the system. The focus becomes making phone calls, setting meetings and sending proposals, rather than generating meaningful sales.

I saw a particularly interesting example of this in a manufacturing company. I built a system for a client that tracked warranty claims and was used to determine their root causes. The system recorded the original purchase, the return, the reason for the return and, ultimately, the root cause. Was it an error in shipping? A packaging problem? Or was there a flaw in the product or manufacturing process?

The purpose of the system was not simply to count warranty claims. It was to understand why they were happening so that the organization could correct the underlying problem.

The culture of the company was built around modern management practices, KPIs and information systems that supported those practices. The numbers in the reporting system became, in many ways, the reality of the company’s world.

The objective was simple: drive defects down to near zero and it appeared to be working. Warranty claims were decreasing. A few months later, I stopped by for a visit. My contact told me that the previous month they had celebrated a near-perfect report.

They had achieved the impossible. She would have told me about it earlier, except there was a problem. Someone had been sick and had not processed the warranty claims. The company had not suddenly achieved near-zero defects. They had simply stopped measuring some of them. Once the outstanding claims were processed, the system reflected reality again.

The lesson isn’t that KPIs are bad. Quite the opposite. Good KPIs are essential to managing an organization. The lesson is that we should never confuse the measurement with the thing being measured. A KPI is our best attempt to create a useful representation of reality. It tells us something about the organization, but it is not the organization itself. And whenever a KPI becomes a target, people will naturally adapt their behavior to the measurement. This is why KPI design matters. It is also why the systems behind those KPIs matter.

And now we are introducing another layer of complexity: AI. We are increasingly using AI to help design systems, analyze information, identify patterns and make decisions. AI can make our organizational models far more sophisticated, but it can also make a bad model much more convincing.

A beautifully designed dashboard can still be measuring the wrong thing. An AI system can analyze thousands of warranty claims, but if the underlying data is incomplete, the AI may simply become very good at explaining an incomplete version of reality. The question, therefore, is not simply: “What does the data tell us?” We also need to ask: “Does the data actually represent what is happening?”

KPIs are measurements. Systems are models. AI is increasingly becoming the engine that interprets those models. We need to use all three wisely. Because ultimately, what we measure is what we pay attention to, and what we pay attention to is what we tend to get.