Reading operations case studies metrics without fooling yourself

Somebody hands you a figure from another team, another company, or a conference talk. You cannot audit it and you have to decide what to do about it.

You can do more than accept or dismiss it. A reported number has a small number of parts, and asking for those parts is quick, polite, and usually decisive. This page is about which parts to ask for and what the answers mean.

What to take away

The five parts of any reported figure

The numerator. What was counted, defined tightly enough that two people would count the same cases. "Complaints" is not a definition; complaints logged in a named system by a named route is.

The denominator. What it was counted against. This is where most misleading figures live, because a rate falls when the bottom grows and nothing about the top has changed.

The period. Start and end, and whether either is unusual. Also whether the period was chosen before or after the data was seen.

The population. Which cases, teams, or products are included, and what was excluded. Every exclusion is defensible and every exclusion moves the number.

The comparison. What the figure is being contrasted with, and whether that comparison was measured the same way. A before-figure produced by a different method is not a baseline.

Ask for all five in one message. Most people can answer in a few lines, and the ones who cannot have told you something useful about how the number was produced.

Why the counts matter more than the rate

A percentage compresses two facts into one and discards the scale. Some invented illustrations of the same reported drop:

All four are reported identically. Three of the four do not support the conclusion the phrase invites. This is why the counts are the first thing to ask for and why a refusal to supply them is significant.

The same trap applies to any figure quoted without the underlying frequency, which is the general problem called the base rate fallacy: a rate detached from how often the thing happens at all cannot be interpreted.

Reading a chart you were handed

A chart like the one above shows something a summary figure cannot: how much the measure moves anyway. That is the first thing to look for, because a difference smaller than the ordinary movement between points is not a finding.

Four things to check on any operational chart.

Where the window starts. Extend it backward in your head. If the series was already moving in that direction before the intervention, the intervention is not the story.

Whether the axis begins at zero. It often should not, and that is legitimate. What matters is whether you noticed, because a truncated axis makes a small movement look decisive.

How the points are aggregated. Monthly points hide weekly variation. Smoothed lines hide everything and are the most common way a chart is made to look like a trend.

What happened to the definition partway along. A step change in a series very often marks a change in how the thing was counted rather than a change in the thing.

When the window was chosen after the fact

This is the hardest to detect and the most common: someone finds the stretch of a series that supports the point and reports that stretch. Nothing is fabricated, yet the result is still not evidence: the pattern was selected, not predicted.

The general form, drawing the target around the hits afterward, is the Texas sharpshooter fallacy.

Two questions catch most of it. Was the period fixed before the data was examined. And what does the series look like either side of the window. Neither is an accusation; both are ordinary requests, and an author who chose the window honestly will answer them easily.

What to do with a figure that survives

Not adopt it. A number that survives this treatment tells you the effect is probably real in that operation, which is a different claim from it being available in yours.

Run a small test with your own baseline, designed so it can come out negative: operations case studies framework.

Before accepting any operational claim, write down alternative explanations, worked through in operations case studies examples. Questions to send the author are in operations case studies questions.

If the figure is one of your own, the standard is higher rather than lower, because internal numbers get checked least. How to construct a measure that will survive this treatment, including what to publish alongside it, is in operating processes metrics.

Common questions

Is it reasonable to ask another team for their raw counts?

Yes, and it is normally welcomed if you say what you are trying to decide. The request that causes friction is one that arrives with no context and reads as verification of someone's honesty.

What if the numbers are commercially sensitive?

Ask for the shape rather than the values: the direction, the number of periods, whether the definition changed. Most of what you need is structural and none of it is sensitive.

How much variation is normal?

That is exactly the question a chart answers and a summary figure does not. Look at how far the points spread before the intervention, and treat any change smaller than that spread as unproven.

Can I use a figure I cannot rebuild?

To decide what to test, yes. To decide what to change, no. The distinction is worth stating out loud in a meeting, because the same number will otherwise be used for both.