I Can See Clearly Now

In 1916 the British Army swapped cloth caps for steel helmets, and the hospitals started seeing more head wounds.

That is the story as it is usually told, and it is worth telling even though the exact admission figures are hard to pin down. The Brodie helmet reached the front in small numbers in late 1915 and in large numbers by the spring of 1916. Shrapnel that would have killed a man in a cloth cap now dented a steel bowl and left him with a scalp wound or a concussion. He went to a casualty clearing station instead of a grave. Dead men are not admitted to hospital, so on the medical returns a life-saving piece of equipment showed up as more injured soldiers.

Picture the staff officer reading that report. Head wounds are up. The new helmets are expensive. Somebody suggests the helmets are giving men a false sense of security. The right answer, that the helmet moved men from one column to a better one, is only visible if you look at both columns at the same time.

Your organization is about to have that meeting. If it is rolling out AI anywhere that touches measurement, such as code review, inspection, maintenance, support or safety reporting, some number on some dashboard is going to go up in month two and look terrible. This post is about how to tell whether that number is a problem or a helmet, and how to make sure the people reading the dashboard know which one before they see it.

The opening and the closing are written for everyone. In the middle you choose your floor: a software team, or a plant floor.

A number on a dashboard looks like one fact. It is really two multiplied together: how much of the thing is happening, and how good you are at seeing it. Improve the second and the number rises, even if the first is falling. The useful question is never just “why did it go up?” It is “who is in the count now who wasn't before?”

There are four ways new people get into the count, and it helps to know them by name.

1. They survived into a milder category. This is the helmet. Seatbelts and airbags cut road deaths and filled rehab wards with people who lived. Neonatal intensive care saves babies born at 23 and 24 weeks, and rates of childhood disability rise because those children are alive to be counted. Cancer and dementia rates climb partly because people no longer die young of other things. The tell is that the severe outcome fell while the milder one rose.

2. The instrument got sharper. Better scanners find small tumors that used to be missed until autopsy. This one has a famous wrinkle. In 1985 Alvan Feinstein and colleagues showed that new imaging was finding hidden spread in lung cancer patients and moving them from early stages into later ones. Those patients were the healthiest people in the later stage and the sickest people in the earlier one, so survival improved in every stage, and nobody lived a day longer. Feinstein named it the Will Rogers phenomenon, after the joke about Oklahomans moving to California and raising the average intelligence of both states. Changing the threshold does the same thing. When the cutoff for diabetes was lowered in 1997, millions of Americans became diabetic overnight without anything changing in their blood.

3. Reporting got easier. Make it easy to complain and complaints go up. The better finding is about what that means. Kyle Welch and Stephen Stubben studied nearly two million internal reports at more than a thousand US public companies, and found that companies whose whistleblower hotlines were used more faced fewer material lawsuits and smaller government fines. A quiet hotline is often a broken one. Aviation and hospital safety programs work the same way: a unit that reports lots of near misses is usually a unit where problems get caught early.

4. The counting rules changed. When the economy improves, people who had given up looking for work start looking again. That moves them from “not in the labor force,” where they are not counted, to “unemployed,” where they are, so the unemployment rate can tick up because things are getting better.

Most technologies trigger one of those mechanisms. AI tends to trigger all of them in the same quarter. It is a sharper instrument: a camera that checks every part, a model that reads every vibration trace, a reviewer that reads every line of code. It makes reporting easier: a chat window instead of a phone queue, a voice note instead of a paper form. It moves things into milder categories: a bearing caught as a work order instead of a seized motor, a bug caught as a red pipeline instead of an outage. And it quietly changes the denominator: more users, more deploys, more parts inspected.

The timing is the problem. The rising number shows up immediately, because that is the instrument working. The falling number takes longer, because returns, claims, downtime and injuries are counted over months, and the hazards you just found still have to be fixed. So there is a window, usually somewhere around month two, where the dashboard is all red and none of the green has arrived yet. That is when the uncomfortable meeting happens, and when good projects get cancelled.

The fix is to read the rising number next to its partner. Here is what that looks like where you work.

Pick the floor you work on, then an example. Each one starts with the number that went up. Press Show what fell to see the partner, who was newly counted, and the one sentence to say to leadership. The three central Wisconsin examples are marked; they come from the industries this region actually runs on.

The software and plant examples are the same pattern in different clothes. A red CI pipeline and an AI camera rejecting a part are both defects stopped at the station instead of at the customer. A rising GitLab backlog and a rising near-miss count are both signs that people trust the system enough to use it. In both places, the number that goes up is the one you can see on day one, and the number that goes down is the one you have to wait for.

One central Wisconsin note. The cheese example belongs to an old local tradition. In 1890 Stephen Babcock at the University of Wisconsin gave dairy a cheap, reliable test for butterfat, and milk could finally be priced by what was actually in it. We told that story in Return to Sender. A new measurement that showed what had always been there built the state's dairy industry. AI in a cheese plant is the same move, a century and a bit later.

None of this means every rising number is a helmet. The argument only works if you are willing to be told it doesn't apply.

1. Finding more doesn't help if finding it changes nothing. After 1999, thyroid ultrasound became a common add-on to South Korea's national cancer screening program. Diagnoses of thyroid cancer rose about fifteen-fold between 1993 and 2011. Deaths from thyroid cancer barely moved. Most of what the screening found were small tumors that would never have caused harm, and the people who had them got real surgery anyway. Researchers call it overdiagnosis. The business version is an AI inspection system that rejects parts no customer would ever have noticed, or a security tool that floods the team with alerts nobody can act on. Detection is only worth it if it leads to an action that changes an outcome.

2. The rate went up, not just the total. This is the honest check for the software floor. AI coding tools multiply output, and output includes bugs. In Ch-Ch-Changes we covered GitClear's finding that duplicated code rose sharply and refactoring collapsed in the AI-assistant era. If issues per active user are rising, or the same kind of bug keeps coming back in different places, the AI is producing problems faster, and the growth story is cover for AI debt. That is why every example in the picker pairs a total with a rate.

3. The number moved because the definition moved. A new threshold, a new category or a new way of counting can make a number rise or fall with nothing real behind it. If the AI rollout arrived alongside a change in what counts as a defect, an incident or a near miss, compare against the old baseline carefully or not at all.

4. The number became a target. Tell a plant that near misses are good news and some near misses will be invented to hit a quota. Reward a team for closing tickets and tickets will be split in two. That is Goodhart's law: a measure that becomes a target stops being a good measure. The paired numbers are the protection. It is easy to pad near-miss reports. It is much harder to fake a drop in recordable injuries.

The practical version is one page long, and it has to be written before go-live, not after. Once the number has already spiked, every explanation sounds like an excuse. Written down beforehand, the same explanation is a prediction, and a prediction that comes true is evidence the system works.

• Name the number that will rise, and roughly by how much. “Open issues will go up as more people use the apps.” “The defect rate will jump in the first month of camera inspection.”
• Say who will be newly counted. Survivors, earlier catches, people who can finally report, or a changed definition. One sentence.
• Name the partner that should fall, and when you expect to see it move. Returns and claims lag by a quarter or more. Say so.
• Track a rate as well as a total. Issues per user, defects per million parts, change failure rate. Totals rise with volume; rates tell you about quality.
• Say what would mean the alarm is real. “If returns haven't dropped by the end of the second quarter, we were wrong.” That sentence is what makes the rest of the page believable.

Or, the short version, as a table to tape to the dashboard:

Go back to the staff officer with the medical returns. Nothing about the report was wrong. Head wounds really were up. What was missing was the second column and a sentence written in advance: head wounds will rise, because men who would have died will now be wounded. With that sentence, the report is good news. Without it, it is a reason to stop issuing helmets.

AI is going to show your organization a lot of things that were always there. Decide now what you want the room to say when it sees them.

This post sits with Out of Spec on the reports an organization filters out before they reach the top, Ch-Ch-Changes on failure statistics with the wrong denominators, Return to Sender on the Wisconsin measurement that built an industry, and Secret Agent Man on the blocked-attack counter you want to see ticking.