Last week I was part of the AIIP Impact Showcase and white-paper launch. Sessions like that are full of numbers: percentages improved, days saved, costs cut. Some of them are real. Some are honest estimates. Some are a hope that got promoted to a fact over three rounds of slide editing. Sitting there, I decided to write down the rules I use for my own numbers.
Three kinds of outcome
Measured. A baseline was recorded before the change, the same thing was measured after, and both are in a document I can show you. This is rare, and it is the only kind I state without qualification.
Reported. The client or the project team stated the result. I believe it, it came from people who were there, but I did not measure it and cannot reproduce it. This is most of what appears in any consultancy's case studies, including ours from the Quantum years.
Projected. Somebody calculated what should happen if the system is used as designed. Useful for planning. Not an outcome.
The sin is not having reported or projected numbers. The sin is presenting them as measured.
How to read a case study
When you read any builder's case study, including the ones on this site, ask four questions of every figure:
- Who measured it, and when?
- What was the baseline, and was it recorded before the work started?
- Is the arithmetic internally consistent? A "50 days to 3 days" claim is a 94 percent reduction, not "around 80 percent." If the percentage and the days disagree, one of them is a slogan.
- Does the number describe a product feature or an organizational result? "Five times faster batch processing" is about the software. "Restructuring finished a quarter earlier" is about the organization. Both can be true; they are different claims.
What I do on this site
Every project story here has a "what changed" list, and under it a provenance line saying where those outcomes come from: the project's own case-study record, a public source, or a measurement. When a figure in the original record was inconsistent, I dropped the inconsistent part and kept the part I can stand behind. Where I only have the scope of what was built and no outcome evidence, I say what was built and stop.
The Telkomsel competency-mapping story is a good example. The project record describes the mapping time falling from around fifty working days to three. I am comfortable stating the days, because they describe an observed change in a specific process. I am not comfortable turning that into a marketing percentage that does not match the arithmetic. So you will see the days, not the percentage.
Why bother
Because the buyer I want to work with is going to check. Founders and operating leaders who have been burned by vendor decks read case studies with a pencil. If they find one inflated number, they discount everything else, including the parts that are true. Labeling provenance costs me a few impressive-sounding percentages and buys me a conversation with someone who now trusts the rest of the page.
For anyone writing their own case studies
Keep a column in your project record for how each outcome was established. Write the provenance on the slide, in small type if you must. And when a number and its percentage disagree, fix it before someone else finds it.
It also means some of the strongest-sounding figures from the original deck do not appear here at all, because the record did not say how they were established. Their absence is the point.
State measured numbers plainly, label reported ones, and never let a slogan outrank the arithmetic.