Case studies · How we check a case
How we check a case.
Every case here is MindPalace’s own output on one business’s data, checked before it is written up. This page says where the numbers come from, what we check, and what the marks in a case mean.
Where the numbers come from
Each case rests on runs of the product whose every response was saved as it came back: the series, each part’s change with its 95% interval, the queries, the product’s confidence and its reasons, and whether a person had reviewed the definition. A tool reads the saved responses, checks them and writes each case’s figures to a file; the pages show only what those files hold.
- The runs of 27 September 2026: KPI Drill-Down, then Deep Analysis in Trend mode, by quarter, on Nimbus’s New MRR and on AdventureWorks’ bike sales to stores.
- The runs of 11 November 2026: KPI Drill-Down and Deep Analysis on Nimbus’s Signups, Churned MRR and Time to First Value; Flow Analysis, Flow with Cohort and Cohort Analysis on Nimbus.
- Flow Analysis, Flow with Cohort and Cohort Analysis ran on the app with fixes not yet released. Until the fixes are released, each case’s record says so.
- Flow with Cohort runs in MindPalace’s engine, but isn’t on the app’s screens yet. The onboarding case shows it, and says so beside it.
- Our follow-up. The bike sales run cut the sales by region and country only. To cut them by bike type and by when each store first bought bikes, we rebuilt the run from Microsoft’s public copy of AdventureWorks, matching it to the cent first, and treated each part as Deep Analysis does. What we worked out is marked as ours.
What we check before a case is written
- Deep Analysis is used only where its quarters match Drill-Down’s. The two count the same metric each quarter; where they disagree, something in one of them is wrong, and we don’t build on it. Churned MRR and Time to First Value fail this today, so their Deep Analysis runs aren’t used: the definition Deep Analysis found for Churned MRR has no churn filter, so it counts every change in subscription revenue, new and upgrades included.
- Checked against an independent count. Over the two years, Nimbus’s New MRR comes to and its revenue lost to churn to . Both match the team’s own count of the same data, made independently of the product’s analyses.
- The parts add up. A split is drawn only if its parts come to the whole change. Where they don’t, it is left out, and the case says so.
- The interval decides the word. A change is said to hold only when its 95% interval stays clear of zero. Where the interval crosses zero, the case says the change might be zero, however large it looks.
- Who defined the metric. A definition a person approved is marked as theirs. One the machine matched on its own is said to be unreviewed, in the app’s own words.
- The product’s own warnings. Its confidence and the reasons it gives are quoted, low as well as high.
- The product’s AI is quoted, not believed. Its summaries and proposals are marked as its words and never used as evidence. Where it misreads, the case says so: it read Nimbus’s newest signup months as a "severe retention cliff", but those months haven’t happened yet.
- Unfinished months are left out. Cohort Analysis returns 0 for a month that hasn’t happened yet; the cases leave it blank, never 0.
- Decisions are proposed until a person takes them. Each case ends in a proposed decision, drawn dashed. Brass marks only what the record shows a person decided.
- What the data is. Every case names its data and says whether it was generated, is a public sample, or is a customer’s.
What the marks mean
The kinds of statement in a case, told apart by form. A figure carries at most one mark; plain means the run returned it.
Observed
What the run returned, as the record has it. The default, so it carries no mark.
Calculated
Worked out from figures the runs returned, such as what churn took for every $100 of new revenue. The working is in the case’s full analysis; the double rule is the total’s.
Checked against a source
Confirmed independently: an independent count of the same data, the parts adding up, or the product’s own definitions.
might be zero
Uncertain
Still open: a change whose 95% interval crosses zero, or a figure that rests on one.
the joins follow
Ours, not data
Pencil: a note of ours, such as where a query is cut short. Never a figure.
"severe retention cliff"
Written by the product’s AI
The product’s own summaries and proposals, quoted as they are: written after the numbers, never checked against them.
approved
A person decided
Brass, and only where the record shows a person decided: here, approving what a metric counts.
Proposed decision
Proposed, not taken
What we’d decide from the evidence, dashed and never brass, because no person has taken it.
The data behind these cases
Nimbus is a company the team generated for demos, with subscriptions on Starter, Pro and Enterprise plans. Its data describes no real company: what holds in it is a fact about that data, not evidence about a market.
AdventureWorks is Microsoft’s sample database of a bicycle maker. Its figures, and its dates, are the sample’s own; Microsoft’s public copy moves the dates later, and for our follow-up we moved them back.
No case here uses a customer’s data yet. A customer story will need the customer’s figures, their consent, and a person who decided something.
Corrections
When a case turns out to be wrong, we correct it on its page and say what changed and when.