Agents Got a Harness. Data Needs a Different One.

A metric called Trial-to-Paid Conversion Rate was being computed as a raw count. The SQL ran. The chart rendered. Nothing crashed. The number was just not the thing it claimed to be.

We caught that one in our own system. The unsettling part was how ordinary it looked. No error, no warning, nothing on the screen to separate it from a number that was right. It would have gone into a board deck, and nobody in the room would have known.

That is the shape of the most dangerous analytics errors. They do not announce themselves. A wrong query can run perfectly, a wrong metric can render beautifully, and a wrong number can drive a real decision without once looking wrong.

This is the problem we have spent the last year on, and it is why a data team needs what the agent world has started calling a harness, and why the data harness a team actually needs is not the one everyone is currently writing about.

A harness assumes the world pushes back

Start with what a harness even is. The word comes from the people building coding agents, where it has settled into a clean equation: an agent is a model plus a harness. The model reasons. The harness is everything around it, the runtime that picks tools, manages context, enforces guardrails, and checks the work. The term caught on because of the result behind it. One team moved a coding agent from thirtieth to fifth on a public benchmark without changing the model at all, by improving the harness alone. The harness, not the model, is what makes an agent reliable.

A coding harness has a few standard layers: tool orchestration, verification loops, guardrails, observability. Underneath all of them sits one assumption so obvious nobody states it. The environment tells the agent when it is wrong. Types fail. Tests fail. The compiler complains. A build breaks. The verification loop works because there is something to verify against. Run the code, read the error, try again, converge. The harness is a machine for converging on the right answer, and it can be, because the world keeps score.

We have been making a quieter version of that claim about analytics. Point a model at a warehouse with no structure around it and you get confident nonsense. Add the structure and the same model gets things right. The model was never enough by itself. The catch is that the harness everyone is writing about was built for code, and code has something a warehouse does not.

Data has no compiler for meaning

Point that same machine at a warehouse and the floor falls out.

It is true that the SQL compiles, the dbt tests pass, and the chart renders. None of that checks whether the number means what it claims to. The warehouse can compile a query. It cannot compile intent.

Data has no compiler for meaning.

There is no build to break, no test to go red, no type error for a number that computes perfectly and means the wrong thing. The agent produces an answer, the answer looks exactly like a right one, and nothing in the environment objects. The single assumption a coding harness is built on is the one thing a data team does not get.

You do find out, eventually. Finance complains, or a stakeholder challenges the number, or the board asks where it came from. But that feedback is late, social, and manual, and by the time it arrives the decision has already been made.

That is why a data harness cannot be a coding harness with the nouns swapped. It has to be a different machine. A coding harness exists to help the agent converge on the right answer. A data harness has to exist to make the wrong answer impossible to miss.

Three places analytics agents go wrong

Before you can catch a wrong answer you have to know how they happen. Sort the wrongness and it falls into three buckets. The agent picks the wrong entity, mapping a loose business term onto the wrong table or metric or grain when several plausible ones exist. It runs on stale context, because the business moved and the definitions did not. Or it fails to retrieve, where the correct answer sits in the warehouse, properly described, and the agent simply never finds it.

These are the causes. The number at the top was the first of them, a metric mapped to the wrong shape, a rate computed as a count. Any data person has shipped every one of these. The grain that exploded and doubled revenue, the denominator that quietly dropped a segment, the time window that was off by a day, all of it traces back to one of these three.

All three produce the same artifact: a confident, plausible, wrong answer that announces nothing. And all three yield to better context, which is the whole reason context is the lever everyone is pulling on. When Anthropic's data team pointed Claude at their warehouse with no extra structure, it answered fewer than 21 percent of their analytics questions correctly, and giving it raw access to thousands of their own prior queries moved that number by less than a point. What carried it past 95 percent was context, built deliberately. The model was not the bottleneck, which is the harness lesson again, in a second domain. We pulled apart the governed layer they built in a separate post.

So context matters, enormously. But it is worth being precise about what context buys. It makes the agent more likely to land on the right number. It gives you no way to know, on any single answer, whether it did. Context is necessary. It is not sufficient. That gap is the data harness's whole job.

A data harness is a different machine

So what does the machine actually do. A few decisions make it up, and a data team can treat them as a checklist for its own harness without copying anyone's code.

It starts with the boundary. The model should plan an analysis. It should not do the arithmetic. The moment you let a language model carry the math you lose repeatability and invite a miscount with no tell at all. Keep the model on intent and structure, run the computation in deterministic code, and at the very least the same question produces the same machinery every time. We made the full determinism argument here.

Then there is meaning. The agent knows what a column contains. It does not know what the column represents, and it will guess the difference from the column name if you allow it. So you decide, on purpose, how a metric binds to its meaning, and you decide what happens when it cannot bind. The honest behavior, and the rare one, is to refuse to certify. Not to refuse the question, exploratory work has to start somewhere, but to refuse to present an ungoverned number as if it were decision-grade truth. A system that says it cannot anchor this number to a governed definition is worth more than one that guesses and prints the guess in the same font as the truth. This is the decision we built grounding around. Cartographer learns what each column actually means from how the warehouse is queried, and a metric only goes live once it binds to a governed definition a person has approved. The model proposes the mapping. It never gets to declare it.

Before any query runs, something has to check it. Structural checks are easy and you should have them. The hard and valuable check is semantic. This is the class the Trial-to-Paid number at the top belonged to: a rate computed as a count, structurally fine and semantically a lie. Every team makes a decision about that class of wrongness, usually without realizing it is making one. The only honest behavior is to mark the number as untrustworthy rather than let it render clean.

Then disagreement. When two ways of computing the same thing produce two answers, the easy path is to pick one and present it cleanly. The honest path is to surface the split, because two methods reaching two numbers is information, and hiding it manufactures a confidence the data does not support.

Provenance comes next. If you cannot trace a number back to its source, its filters, the exact query that produced it, you cannot defend it and you should not trust it. That is the difference between a number you can stand behind and one you are only repeating.

And underneath all of it sits the decision the rest depends on, which is what the system does when it is not sure. This is where most analytics tools quietly fail, because a low-confidence answer renders identically to a high-confidence one. The defining job of a data harness is to make sure that is never true. Every answer should carry its own status. Whether the fact underneath it was declared by a human, inferred by the system, or actually verified is not metadata to bury in a tooltip. It is the most important thing on the screen, and it belongs where the reader cannot look past it.

A harness is never finished

There is a last property of a harness that is easy to miss, and it is the one we would not skip. A harness is not something you build once and own. The failure surface keeps moving. The business changes, definitions drift, and the context that made an agent accurate last quarter goes quietly stale, the accuracy bleeding out from under you while every individual answer still looks fine.

This is not a hypothetical risk. Anthropic reported that their own accuracy fell from about 95 percent to about 65 percent in a single month from exactly this, and they held it up with a code-review hook that flags any change to a reporting model that does not carry the matching change to its context. Roughly 90 percent of their data-model changes now ship with the context change in the same diff. That fix is discipline, a rule enforced on every change, and discipline is the thing that holds in a team that rigorous and erodes nearly everywhere else.

So the direction is not hard to see. A data harness should treat its own staleness the way it treats a wrong answer. It should make its decay visible instead of waiting for someone to notice the numbers sliding. The principle that governs the answers has to govern the harness itself. When the harness changes, that change should be as visible and reversible as the answer it produces, because a harness that quietly rewrites itself has reintroduced the exact problem it exists to prevent, one level up.

The job

This is not a UI feature. It is a set of product decisions, and we are laying them out as decisions because the answers are not obvious and most teams reach them by accident. The agent is the easy part. The model is already good enough to plan an analysis and read what a question is really asking. It was never good enough to be trusted as the thing that computes the number, checks it, and signs off on it. Those are different jobs, and the harness is what keeps them separate. Take the boundary alone. A model that hands you a number and a system that recomputes that number the same way every time look identical in a demo, and could not be more different the first time someone asks how it was calculated. What is hard, and what will separate the data teams that come to trust their AI from the ones that get quietly burned by it, is everything around the model that decides what happens when it is wrong. That is also where the analyst who owns meaning earns the seat the lookup service is losing.

In coding, harness engineering is becoming its own discipline because everyone learned the model was not the bottleneck. Data teams are about to learn the same thing, with higher stakes, because nothing in the warehouse catches the miss for you.

In a domain with no compiler, the job is not to make the agent sound more confident. The job is to make uncertainty visible, make provenance unavoidable, and make silent wrong answers structurally harder to ship.

A data harness cannot guarantee truth. It can stop unverified answers from masquerading as truth.

Coding agents got a harness. A data team needs a different one, and that is what we have spent the last year building.