Operations case studies mistakes: the traps and the way around them

This page is about the write-ups your own organization produces, not the ones you read from outside. Internal accounts are trusted more, checked less, and cited for years, which makes their defects more expensive.

Every mistake below is an omission rather than a falsehood. That is what makes them survive review: nothing in the document is untrue.

What to take away

1. Written from the ending backward

What it looks like. A clean narrative in which each step follows from the last and arrives at the result.

What it does to the reader. It hides the branch points. The reader cannot tell which decisions were fine judgment calls and which were forced, so they copy all of them with equal confidence. Knowing the outcome makes the path to it look inevitable, an effect general enough to be named hindsight bias.

The fix. Keep the options you rejected in the document, with the reason. Two lines each.

2. No baseline, or a baseline described in words

What it looks like. "Handling times were a problem" followed by a precise figure for afterward.

What it does to the reader. It makes the improvement unquantifiable and unfalsifiable. It also invites the reader to assume the before-state was as bad as their own, which it may not have been.

The fix. One measured before-figure, with its definition and period, even if it is rough. A hand tally beats an adjective.

3. The person is doing the work the method is credited with

What it looks like. A method described in detail, with one exceptional individual mentioned in passing as having run it.

What it does to the reader. They adopt the method without the person and get a different result. This is the most common reason an internal practice does not spread.

The fix. Ask explicitly what the person did that is not in the written method, and put the answer in the document. Usually there is something and usually it is the whole thing.

4. The failed attempts are missing

What it looks like. One approach, which worked.

What it does to the reader. It conceals the cost. If two approaches were tried and abandoned first, the real price of the result includes them, and a reader budgeting from the document will underestimate by a factor.

The fix. A short section listing what was tried and dropped. It is the section people are most reluctant to write and the one most often quoted back gratefully.

5. The context that made it work is unstated

What it looks like. A practice described as though it were portable.

What it does to the reader. They apply it in a setting where the enabling conditions are absent. Volume, skill mix, tooling, and how much slack the team had are the usual conditions, and none of them feel worth mentioning to the author because they are just the background.

The fix. Answer one question in the document: what would have to be true elsewhere for this to work. The general problem of whether a result travels is treated as external validity.

6. Only the successes get written up

What it looks like. A folder of accounts, all of which describe things that worked.

What it does to the reader. It distorts their sense of the base rate. If nine attempts produced one success and only the success is documented, the archive says this approach works, and the archive is wrong.

The fix. Write up the ones that did not work, briefly, in the same place. Three paragraphs is enough and the discipline matters more than the length.

7. A single case reported as a rate

What it looks like. "Reduced by a third", from a comparison of one month against one month.

What it does to the reader. It supplies false precision. At most operational volumes a single period comparison cannot distinguish a real change from ordinary variation, and the percentage carries an authority the underlying count does not support.

The fix. Report the counts alongside any rate, and say how many periods were compared. Where the volume is low, describe the cases instead of computing a rate from them.

8. No statement of what is still unexplained

What it looks like. A tidy account in which every observation is accounted for.

What it does to the reader. It stops them asking. An account that names its own loose ends invites checking; one that names none is accepted whole, including the parts that were never tested.

The fix. A closing line saying what did not fit. It costs one sentence and it is what makes the rest credible.

Writing one that holds up

Four habits, and none of them add much length.

Write the prediction before the result. Even one line, dated, saying what you expect. It is the only protection against reconstructing your own reasoning afterward.

Keep the numbers with their definitions in the same paragraph. Separated by a page, they come apart within a month, and the figure travels without the definition. Why that matters, and what a definition change does to a comparison, is in operating processes metrics.

Name the conditions. Volume, team size, experience, and what else was happening at the time. A reader in a different setting can then do their own translation instead of assuming there is nothing to translate.

Say what you would do differently, separately from what you would recommend. These are different lists and merging them loses the more useful one.

What a good internal account is actually for

Not persuasion. Its job is to let a team elsewhere decide whether to run their own small test, and to design that test better than they would have from scratch. That is a much lower bar than proving something, and it is achievable.

Judged that way, the sections that matter are failed attempts, conditions, and the unexplained remainder, and those three get cut for length.

The reading side of this, applied to accounts from outside, is in operations case studies. The way to turn any of it into a test you can run is in operations case studies framework.

Where an account is being used to justify a change to how people work, the sequencing of that change is a separate skill and sits in change management.

Common questions

Is it realistic to write up failures in an organization where that is career damage?

Not always, and pretending otherwise is not useful. Where it is not possible, write the conditions and the alternatives at least, since those two carry most of the transferable value and neither requires admitting anything.

How long should an internal write-up be?

Two pages. Longer ones get skimmed for the conclusion, which is the part that transfers least.

Should the person who ran it write it?

They should supply the material, and somebody else should ask the questions. An author writing about their own work will genuinely not see the enabling conditions, because to them those conditions are simply the situation.

What about a write-up requested by leadership for a decision already made?

Then it is a briefing rather than a case study, and it is worth being clear which you are producing. The dangerous version is a briefing filed where case studies live, because a year later nobody remembers which it was.