We Were Never Measuring What Users Wanted

Ask a room of data analysts whether AI is coming for their jobs and you get a nervous laugh. The real question beneath the laugh is whether AI will replace data analysts. It deserves a straight answer, not reassurance. Here is mine. The profession is not going away, but the job is about to split, and the half that survives is harder and more consequential than the half being automated. To see why, start with the problem every data team is pouring itself into right now.

The problem everyone is solving

Right now almost every data team I talk to is working on the same thing. Get a language model to turn an English question into a correct SQL query. "What was net revenue retention in EMEA last quarter." The model reads the question, writes the query, returns the number. It is a genuinely hard problem. The schema is ambiguous, the joins are unobvious, and the same word means one thing to Finance and another to Growth.

It is also, and I want to be precise about this, a solvable one. We know because we built the governed version of it ourselves. The thing that makes it solvable is not that the model is clever. It is governance. Once you pin the definitions, the semantic layer, the metric logic, you have manufactured a ground truth, and now the query is checkable against it. Revenue means the contracted figure, not the billed one. EMEA excludes the UK reseller deals after the reorg. With those pinned, a generated query is either right or wrong against a standard you defined, and you can catch it when it drifts. Intent is linguistic. Math is deterministic. Keep the model out of the math and the SQL gets tractable.

This is the architecture two of the best model labs in the world both landed on. Neither OpenAI nor Anthropic, by their own published accounts, pointed a raw model at a warehouse and trusted the answer. They governed the metrics first. So hold onto why that problem is solvable, because it is about to matter. It is solvable because you can build it an answer key.

We were always inferring

Step back from the query, because the harder thing is hiding right behind it.

For as long as I have worked around data, the job has rested on a quiet assumption almost nobody states out loud. We never had access to what users wanted. We only ever had access to what they did.

A click is not a desire. A session length is not satisfaction. A cart abandonment is not a decision. Each one is a trace, a residue left behind by a person whose actual intent stayed locked in their head. The whole discipline grew up as a way to work around that locked door. We could not ask people what they meant, so we learned to reconstruct it from the marks they left. Funnels, cohorts, retention curves, propensity models. All of it forensic. We read footprints and try to describe the person who walked through.

Plenty of analytics really is measurement. Revenue booked, headcount, units shipped, nobody is inferring a hidden mind to get those. But the part that tries to explain or predict a person is a different animal. The recommendation model inferring what you want from what people like you bought, the churn model inferring your dissatisfaction from your fading visits, the conversion model inferring your purchase intent from how deep you browsed, none of those is measurement. Each one takes behavior and infers a mind behind it. For that whole class of work, the inference was the product. The behavior was only the evidence we could get.

Now a copilot goes into the product, and the user starts telling you what they want. Not in clicks. In words. "I need a gift for my father who hikes, under eighty dollars, here before Saturday." "Does this jacket run small, I am usually a medium but your sizes never fit me." Stated intent is not new. We have had it for decades in surveys, support tickets, search boxes, reviews. What is new is that it now arrives continuously, inside the product, attached to the action, at the scale of every session and in the customer's own unprompted words. The locked door appears to open.

Two witnesses now, and they disagree

It is tempting to read this as the end of the guessing. The measuring era finally begins. I do not think that is what happens. The inference does not disappear. It moves.

Watch the interface itself change, because the data changes with it. In e-commerce the customer was a trail you pieced together after the fact: search terms, then clicks, then the whole funnel from first impression to abandoned cart. Now, where these agents take hold, she skips the funnel and just tells one what she wants in a sentence. The trail is starting to turn into a transcript. It is early, and it is uneven, but it only runs one way.

That changes the analyst's job at the root. A large part of the old job was to turn the trail into a defensible number. Why do carts get abandoned? She cut the cohort, found the drop-off step, handed merchandising a figure they could act on. Hard work, but it had a floor: she was counting things people actually did, and in the end the shopper either bought or she did not. The number could be checked.

The new job starts from language, and language cannot be counted until it is interpreted. Ten thousand shoppers a day now tell the agent, in words, what they want and why they leave. The analyst's job is to read all of it and decide what is actually true, because the words are not the truth either.

Six thousand of them typed "too expensive." But the shoppers who actually went quiet mostly said nothing at all, and when the team shortened delivery times without touching a single price, most of the price-complainers came back. The stated reason was price. The behavior answered to convenience. The words say one thing and the behavior says another, ten thousand times over, and she has to decide what they meant and then build the number anyway. Does the report read "we are losing them on price," or something truer and far less clean? Whatever she writes, merchandising reprices around it.

And here is the difference everything else hangs from. The old number often had a native error signal. Predict that a shopper is gone for good, and when she comes back and buys, the outcome tells you that you were wrong. Behavior can eventually grade you.

The new number has no equally direct signal. Read ten thousand stated reasons, decide they left on price, reprice the catalog around it, and if you misread them, nothing cleanly tells you. Later behavior, experiments and follow-up questions may give you evidence, but no single outcome grades the interpretation. The chart can still look clean, the decision can still look data-driven, and plausible-and-wrong can survive the meeting. Meaning does not throw an error when you get it wrong.

It is the same fact that made the SQL problem solvable, turned over. There you can manufacture an answer key: pin what revenue means and a query is right or wrong against it. Here there is nothing to pin. What ten thousand people meant never becomes an outcome you can check. It is more a spectrum than a switch, some questions you can build most of an answer key for and some almost none, but the ends are far enough apart to run the whole job on. Where you can build the key, the work is cleanly checkable, governable, automatable. Where you cannot, you can still check and govern and automate pieces of it, just never with the same precision and never safely closed without a person reading the result. That gap is exactly where the analyst now lives.

Will AI replace data analysts?

Notice what that uncheckable number is, before you despair of it. A number with no stable error signal cannot be safely automated end to end. You can absolutely build a model that reads meaning and guesses, we already automate plenty of subjective work that way, on weak labels and human feedback. What you cannot do is turn it loose and trust it, because nothing reliably tells it when it is wrong. So it runs with a person in the loop, or it does not run safely at all. The very thing that keeps it from closing into hands-off automation is the thing that makes it easy to be confidently wrong. It needs judgment, which is to say it needs a person.

So, will AI replace data analysts? Some of them, yes, and that is the part the comforting version leaves out. AI does not replace the analyst so much as split her. It promotes the one who can own meaning, who holds the words against the behavior and stakes a read she can defend. And it comes for the one who stayed a lookup service, passing through numbers she never interpreted. The half of the job that survives is the harder half. The half that gets automated is the half that always could have been.

That gap, where reality gives the analyst no clean error signal, is the work, and it is what we are building a layer around: construct the strongest available signal from conflicting evidence. Surface the uncertainty instead of laundering a guess into a clean answer. Let a model propose a reading and leave a person to sign off on it. Flag the conflict instead of quietly resolving it. Not a machine that quietly decides meaning for her, but one that tells her, loudly, when the evidence may not support her interpretation.

Analytics is becoming, openly and at last, what it always quietly was for any question that was ever about a person: a way of inferring what is in someone else's mind. The evidence set did not change so much as widen, from what people did to include what they now say. That is richer evidence, not necessarily truer. Reconciling the two is the new job. It is not the end of the guessing. It is the guessing finally admitting what it is, and handing the hardest part of it to the people who were always doing it anyway.

---

Further reading: why LLMs should never calculate your churn rate, the Human API problem, and how OpenAI and Anthropic built the governed layer in-house.