Three minutes to a draft, and the unmeasured hours to verify it.
Three minutes instead of thirty. That’s the measurement at the centre of nearly every AI business case I’ve seen, and it’s the wrong number.
The best version of it came with its own refutation attached. Reckitt’s Imteaz Ahamed, talking about generative AI in CPG, needed to explain multi-armed bandits to a senior executive. He asked ChatGPT for the explanation, then for a Midjourney prompt, then generated the image. Three minutes. Unaided, about thirty. A very new analyst, twenty.
Then, in the same breath, the standard he actually applies to the output: have you ever asked a junior to pull data? Good starting point, and then you have to work out what’s in there.
Three minutes is time to a draft. Not time to something you’d put in front of a customer.
I went looking for someone who disagreed and couldn’t find them. Bryan Gildenberg, hosting that same conversation, makes it an onboarding question - you wouldn’t let a new employee act “right out of the gate without adult supervision.” Eugene Yan describes a ladder of verification, cheap and deterministic before expensive and judgement-based, and argues elsewhere that an LLM-as-judge won’t save the product because the failure is in the process around the judge, not the judge. Anita Watkins of the Insights Association bounds synthetic respondents by stage rather than by quality. SAS’s Charles Chase has built the human check into the product.
Different vocabularies, one position: the check is mandatory, and the live argument is only about making it cheap. Which is where the money question sits, and none of them goes there.
Two columns, not one
| Time to output | Time to verified output | |
|---|---|---|
| Slide on multi-armed bandits | 3 min | ? |
| your use case |
Every business case I’ve read fills in the left column. I’ve never seen one fill in the right.
The arithmetic is worse than it looks. Say review takes twelve minutes. Against a thirty-minute baseline that’s a 50% saving, not the 90% on the slide. And it doesn’t scale, because reviewing ten outputs isn’t ten times easier than producing them - the expensive part of review is loading the context, and you reload it every time. Somewhere that curve bends. Nobody has published where.
Which is also the honest limit of everything below. One measurement exists in all of this and it covers a single slide. Direction, not magnitude. I wouldn’t let anyone size a saving with it.
Three ways to make a check cheaper
Order the checks by price. Deterministic tests before judgement-based ones.
A deterministic test has one right answer and no opinion about it. Does the JSON parse. Do the columns sum to the total. Is that SKU in the master list. Is the date inside the reporting window. It runs in milliseconds, costs nothing to run again, and gives the same verdict every time.
A judgement-based check needs someone to form a view. Is this summary accurate. Is this recommendation sensible for this account. Does this read like us. That’s a person’s time, or another model call plus a person spot-checking the model - and either way it’s the expensive column.
Run the cheap ones first and the expensive ones only ever see what survived. That lowers the cost of a mandatory check rather than removing it. And when a judgement-based check fails, the fault is almost never the tool doing the judging - it’s the process around it: labelling, hypothesis, experiment, all of it human work someone pays for.
Draw the boundary by consequence, not accuracy. Use the machine where being wrong is recoverable; keep a person in the loop where it isn’t. Much cheaper to answer than “is the model good enough”, because reversibility is a property of the decision and you already know it before you start.
Make the review the training signal. A planning system that learns from the planner’s own historical overrides doesn’t take the planner out of the loop - the loop is the input. Review stops being overhead the moment it compounds.
Where the money actually goes
This is the part I care about, and it’s mine rather than anyone’s.
If review is mandatory and review costs money, value doesn’t accrue where the model is faster. It accrues where the review gets cheaper. That moves the investment off the model and onto the scaffolding around it:
- Deterministic tests first. Anything a regex, a schema check or a sum can catch should never reach a person.
- Narrow output instead of free text. A structured field is checkable in seconds. A paragraph has to be read.
- A traceable link to the source. Most review time goes on working out whether a claim is real. If the output carries its provenance, that time collapses.
Follow it through and you get an awkward conclusion. A model 5% better on a benchmark that hands you free text loses to a worse model that hands you a checkable structure. That is not a popular thing to say to a vendor, and I’ve never heard it said in a procurement meeting.
The junior analogy is doing too much work
Everyone reaches for it, including me. It’s an analogy, not a measurement.
Nobody has shown that a model’s errors are distributed like a junior analyst’s. My suspicion is that they’re rarer and more dangerous - they arrive with more confidence and without the tells. The hesitation, the hedge, the “is this what you meant”. You learn to read a junior within a month. There’s nothing to read here.
Which matters for the second column, because a check calibrated on how often a junior is wrong will be calibrated wrong.
The test
One question, and it fits in any business case review:
What’s the second number?
If there isn’t one, the case hasn’t been calculated. It’s been sketched.