What Is the AI Actually Allowed to Decide?

Before you buy an AI product, one question will tell you more than almost any number in the deck: what is the AI actually allowed to decide on its own, with nothing downstream checking the decision?

Most AI vendor claims tell you what the system does: how many agents it runs, how much of the workflow it automates, how quickly it returns an answer, what models it uses. Those things can matter. They do not tell you what the system is safe to trust.

AI activity and AI authority are not the same thing.

Activity is how much the system does. Authority is which of its decisions everything downstream simply believes. A system can automate almost its entire pipeline and still be dangerous if the model is given unchecked authority over a decision everything else depends on.

Consider something as basic as Revenue. An AI chooses which definition to use. A query engine turns that choice into SQL. The warehouse aggregates it. A dashboard renders it. Everything downstream can behave perfectly and still produce the wrong answer because the original decision was wrong. Deterministic machinery can make a decision repeatable. It cannot make the decision correct. The full argument for that split is its own post.

A disclosure before the rest: this is a guide to evaluating an AI vendor, written by an AI vendor. Read it with that in mind. We have spent the last year measuring our own product with frozen benchmarks and audited answer keys, and some of the metrics we started with did not survive that scrutiny. Each section below takes a common vendor claim and turns it into the question that actually discriminates. At the end is a test you can run on any AI vendor. Run it on us too.

"Our agent can do everything"

Whether a wide-open agent is impressive or reckless depends on one property of its environment: does the world push back?

A coding agent can be handed enormous freedom because code pushes back constantly. The compiler complains, types fail, tests go red, the build breaks. A wrong attempt announces itself in seconds, so the agent can afford to attempt almost anything. A data agent lives in the opposite world. SQL can execute perfectly while meaning the wrong thing, and nothing in the warehouse objects. We have made this argument in full before: data has no compiler for meaning. The same lens applies anywhere an agent acts. Drafting the support reply is cheap to get wrong. Issuing the refund is not.

So "our agent can do everything" is not a capability claim to admire. It is a risk posture to interrogate. The right amount of autonomy is set by how expensive silent failure is, and in analytics, silent failure ships in a board deck.

"We have guardrails"

Guardrails is a word that means everything and therefore nothing. Make it concrete with one question: what does the system do when it is not sure?

There are only a few possible answers. It can guess. It can retry until something scores well enough. It can quietly take the highest-ranked interpretation. It can disclose the ambiguity. It can refuse to certify the answer. It can ask a person. Most products do one of the first three and describe it in the language of the last three.

In our product, a metric that cannot bind to a governed definition does not render like a metric that can. It is marked ungoverned, and the system says it cannot anchor this number to an approved definition rather than printing a guess in the same font as the truth. That behavior costs us demos. Prospects want the number. It is also the behavior we would fight hardest to keep, because of what it prevents: a low-confidence answer that renders identically to a high-confidence one.

A trustworthy system is not one that never becomes uncertain. It is one where uncertainty cannot masquerade as certainty.

"We have a semantic layer"

Good. So do we. The question is who keeps it true.

A semantic layer is correct the day it ships. Then finance changes how revenue is treated, a product gets reclassified, a new channel appears, an owner leaves, a one-off exception hardens into policy. The layer notices none of this on its own. A governed definition that was right in January and untouched since is not governed. It is embalmed.

We have watched this happen to ourselves, in miniature. One warehouse in our benchmark scored well for weeks partly on stale mappings, definitions our system had linked once and kept leaning on. A catalog rescan retired them and the score fell. That stung, and it was the honest outcome, because some of those earlier wins were never real. Nothing had objected in the meantime; nothing checks a definition that already looks settled. That is drift inside a system whose entire job is definitions, run by people watching for it. Now imagine it inside a tool nobody audits.

So the buyer question is not "do you have a semantic layer." It is: when the business changes, what carries that change into the layer, and how would anyone notice if nothing did? We wrote about why a semantic layer is correct exactly once, and why the record of meaning has to live inside the system of work rather than beside it.

"We use the best model"

The shortest section, because the market is settling it for us. Every serious vendor rents the same handful of frontier models, and the gap between them narrows every quarter. When OpenAI showed its internal data agent, the interesting part was not the model, it was the context architecture underneath it. When Anthropic described its internal analytics setup, a frontier model scored poorly against their own warehouse until a governed semantic layer was built around it.

The model is rented. The machinery around it is the thing a vendor actually built, and it is the only part of the product that can be a moat.

"We are 95 percent accurate"

Two questions dissolve most accuracy claims. Ninety-five percent of what? And what happens in the other five?

The first question sounds pedantic until you watch one system score against two environments. We benchmark our grounding engine, the part that decides what each business metric means in a customer's warehouse, against multiple warehouses with audited answer keys. At one point the same pipeline, under an identical scoring rule, scored 21 percent on one warehouse and 61 percent on another. Same product, same code, same week. Either number quoted alone would have been true and misleading at once. Clean schemas flatter a system. Messy ones tell the truth. An accuracy number without the eval set behind it is a number without a denominator, so ask what the set looks like, whether it resembles your mess or their demo, and whether it is frozen or quietly retuned between releases. Ours was frozen and audited, and it still lied to us four times.

The second question matters more. Two systems can both be right 95 percent of the time and be entirely different products. One is wrong five percent of the time and confidently ships all of it. The other catches most of that five percent as ambiguous, declines to certify it, and shows a person what it found. Same score, completely different operational risk. This is why we gate our own releases on precision before coverage: of the answers the system commits to confidently, how many are truly right, measured before we ask how many questions it answers at all. A system that says "I do not know yet" a quarter of the time and is almost never confidently wrong is something you can deploy. A system that answers everything and is confidently wrong one time in ten is a liability with good marketing.

One refinement we only learned by scoring ourselves honestly: refusal can be gamed too. A system can post a beautiful precision number by refusing everything difficult. So a refusal only counts when it arrives carrying what the system found: the candidates it weighed, the ambiguity it hit, the reason it stopped. A naked "I don't know" improves the vendor's precision statistic and gives you nothing.

An aggregate score tells you how often a system succeeds. It tells you almost nothing about how it fails.

"There is a human in the loop"

Every vendor can say this truthfully. Somewhere, a human approves something. The useful questions are structural. Where exactly is the human? What can they approve? Does the approval become durable state? Can the AI later overwrite it? And the question almost nobody thinks to ask: what happens when new evidence contradicts an old approval?

That last one stopped being hypothetical for us this summer. In one of our test warehouses, a person had approved the definition of top-line revenue. Weeks later, a fresh run of our own resolution pipeline produced a different definition for the same metric, one that computed a number roughly four times too small, and stood ready to mark it high-confidence. The only thing standing between that wrong definition and the top line was the earlier approval. The approval did its job.

It also did something we did not expect. Because the system treated approved as final, it had stopped re-examining that metric entirely. Had the approval been the stale side, and the fresh evidence right, nothing would ever have caught it. The mechanism that blocked the regression was also the mechanism hiding it.

That incident taught us the real requirement, and it is sharper than "human in the loop." An approval has to protect a decision from casual overwrite without making it immune to contradicting evidence. Those are two different guarantees, and most systems that deliver the first quietly destroy the second.

When evidence and approval disagree, the honest behavior is neither "the AI wins" nor "the human wins by default." It is to surface the conflict, plainly: this definition is approved, and current evidence contradicts it. A person decides, with both in view.

Governance is not the presence of a human. It is an explicit, inspectable hierarchy of authority, with a rule for what happens when the authorities disagree.

The test I would actually run on an AI vendor

Do not evaluate an AI data product with a question it can answer. Bring a question with traps in it, and watch its behavior instead of its output.

Give every vendor the same task against realistic data. Something like: show me enterprise revenue last quarter, excluding partners. Then make the traps real. Leave "enterprise" undefined. Have two revenue definitions live in the warehouse. Leave partner classification incomplete. Let one source run stale. If you can manage it, include a definition a human approved months ago that recent data quietly contradicts.

Then watch for six behaviors. Does it guess, or does it ask? Does it surface both revenue definitions, or silently pick one? When it cannot classify partners, does it say so, or does the number simply come out? Can you open the answer's provenance all the way down to the query? If it refuses anything, does the refusal carry what it found, or is it a bare shrug? Then ask the same question three different ways, because a system that returns three confident, different answers has told you everything you need to know.

The easiest way to evaluate an AI vendor is not to ask a question it can answer. Ask a question it should know not to answer.

Models will keep converging, and the demos already look alike. What will not converge for a long time is the machinery this piece has been circling: whether the AI gets to pick the revenue definition everything downstream believes, whether a guess can render in the same font as the truth, whether an old approval can be contradicted by new evidence, and whether anyone would notice the day a definition goes stale. Each of those is a decision the vendor has already made. The demo will not show you which way. The test above will.

That is the part worth paying for. We have placed our bets on every one of these questions, in public and in the product. When you get to us, ask the hard version.