Four common operations claims to question and what else could explain the same result

Four claims follow. They are invented, phrased the way operational write-ups actually phrase things, and each one is followed by the explanations that fit the same evidence.

The exercise is not debunking. In every case the stated explanation might be right. The point is that three or four other explanations fit equally well, and the claim gives you no way to choose between them.

What to take away

Claim one: "We cut average handling time by a third"

What is offered. A new process, introduced in the first month, followed by a measured drop in the average.

What else fits.

The mix of work changed. If simpler cases became a larger share, the average falls with nothing improved. This is the most common explanation and it is invisible unless the result is split by case type.

The definition of the end point moved. If cases now stop the clock at handover rather than at resolution, the measure is shorter by construction.

Cases that used to be handled started going elsewhere. If the difficult ones are now routed to a second team, the first team's average improves and the organization's total does not.

The average was the wrong summary anyway. Operational durations have a long tail, so the mean moves with a handful of extreme cases in either direction while the typical experience does not change at all.

What would settle it. The median and the slowest tenth, split by case type, on a fixed definition, over the same months in the previous year.

Claim two: "The pilot team's output rose by a fifth"

What is offered. One team ran a new practice for six weeks and produced more.

What else fits.

The team volunteered. Volunteers are more capable or more motivated or both, and the result then describes them rather than the practice. This is the single most common flaw in pilot results, and the general form is selection bias.

They were being watched. Attention changes behavior in a way that fades, which is why a six-week gain often does not survive to week twenty.

The comparison period was unusual. Six weeks against six weeks is short enough that ordinary variation produces this size of difference regularly.

Work was taken off them. Pilot teams often get relief on other duties, formally or informally, and the relief does not appear in the write-up.

What would settle it. Assignment that was not voluntary, a comparison group with the same visibility, and a look at the measure three months after the attention stopped.

Claim three: "Defects fell after we introduced the checklist"

What is offered. A count of recorded defects, before and after.

What else fits.

Reporting changed. A checklist arrives with a message about quality, and the same message discourages logging small things. A falling recorded count is consistent with fewer defects and with fewer reports, and the number cannot tell you which.

The starting point was unusually bad. If the checklist was introduced because of a bad quarter, some improvement follows without any intervention, which is the ordinary behavior described by regression toward the mean.

Something else arrived at the same time. Checklists rarely come alone. Training, attention from a manager, and a new supervisor are all common companions and all plausible causes.

Defects moved downstream. If the checklist catches things earlier, the count at the later stage falls while the total is unchanged.

What would settle it. Escapes rather than catches: defects found by the customer, which are much harder to suppress by reporting behavior. A longer baseline would help too.

Claim four: "Since the reorganization, the backlog is down by half"

What is offered. A backlog count, before and after a structural change.

What else fits.

Items were closed rather than completed. Backlog cleanups routinely accompany reorganizations, and a closed item and a done item look identical in a count.

Intake fell. A backlog is a stock, and a stock falls when arrivals slow just as readily as when completions speed up. Nothing in a backlog figure distinguishes them.

The definition of the backlog changed. A new system with different states almost always redraws the boundary of what counts as waiting.

Seasonality. Many operations have a quieter period, and reorganizations are often scheduled for exactly those periods because they are quieter.

What would settle it. Arrival and completion counts separately over a year, on one definition, with any bulk closures marked. A stock tells you almost nothing without its flows.

The pattern across the four

Each claim was a single composite number compared across a boundary, where something else also changed at that boundary. That is the general shape, and once you see it the questions write themselves.

Three checks cover most of it. Did the definition move. Did the population move. Did the flows move, as opposed to the stock. Applied in that order they take an hour and they resolve most operational claims into either something worth testing or something that was never evidence.

Asking what else would produce this figure is not skepticism for its own sake. It lets you act on claims that survive.

To check your own numbers to this standard, see operating processes metrics. To turn a surviving claim into a test you can run, see operations case studies framework.

Using this on your own results

The uncomfortable version of this exercise is running it on a result you produced. It is also the version that matters, because your own results are the ones your organization will act on without asking anyone.

Write the alternative explanations into the document yourself. It reads as weakness for about a paragraph, then as the only account anyone can rely on.

The general reading discipline is in operations case studies, and the ways a write-up goes wrong are in operations case studies mistakes. The eleven questions worth sending the author of any of these claims are in operations case studies questions.

Common questions

Is this not an argument for never believing anything?

The opposite. Listing the alternatives is what lets you believe the claims that survive it, and there are always some. A reader who cannot rule anything out is stuck believing everything or nothing.

How long should this take per claim?

Ten minutes to list the alternatives, and then it depends on which ones you can check cheaply. Most of the time two of the four are checkable from data you already hold.

What if I cannot check any of them?

Then say so and treat the claim as a reason to run a test rather than as evidence. That is not a failure of the exercise, it is the exercise working.

Do these four cover most operational claims?

The four shapes recur, yes: an average that moved, a pilot that outperformed, a count that fell, and a stock that shrank. Almost every operational claim you will be handed is one of those four wearing different clothes.