Your Eval Will Lie to You

For years, the standard benchmark for AI-generated SQL was Spider. By late 2024, the best agents were clearing 90% of it. Text-to-SQL looked close to done.

Then the same research group built Spider 2.0 out of 632 real enterprise workflows: databases with over 1,000 columns, multiple SQL dialects, data living in BigQuery and Snowflake instead of tidy SQLite files. The paper's own agent framework, built on o1-preview, scored 91.2% on Spider 1.0 and 21.3% on Spider 2.0. The official benchmark site reports plain o1-preview at 17.1%. Leaderboard scores have climbed a long way since, as teams tuned their agents against the new test, which is a lesson of its own. But those were the numbers at release, and the gap is the point.

The model was not the source of the collapse. The task was. The first benchmark was not measuring whether AI could do analytics work. It was measuring whether AI could do the benchmark.

Adyen and Hugging Face found the same cliff from a different direction. They built DABstep from over 450 tasks drawn from a real payments platform, and the best agent they measured scored 14.55% on the hardest ones.

If you are building an AI product on top of data, this is not trivia about other people's benchmarks. It is a preview of your own roadmap. AI data agent evaluation inherits this problem in miniature the day you write your first eval set: the eval is a proxy for the job, and proxies drift. I have spent more than a year building AI data systems, and at MindPalace we are building a decision-intelligence platform on a deterministic SQL engine. Our eval system is the reason the product got better instead of just looking better. It is also, on four specific occasions, the thing that lied to us.

The lies follow patterns. Here they are, along with what the best public eval systems get right first, because that part matters too.

The state of the art in AI data agent evaluation is real

The strongest published work on evaluating data agents right now comes from the labs running them internally.

Anthropic wrote up how their internal analytics agent is evaluated. Without procedural knowledge about the data it scored 21% on their evals; with curated skills and a semantic layer, above 95%. They know which layer earned the gain because they ablate one component at a time against a fixed eval set (grep access to their entire dashboard and transformation SQL moved accuracy less than a point), they harvest every stakeholder correction as a candidate eval, and no domain launches until its slice clears a threshold (around 90% at first).

OpenAI's in-house data agent tells the same story: context around the model, corrections that persist, evaluation as a continuous practice. We covered what it teaches about AI-native BI separately. All of it is correct, and if you are building an AI data product you should do all of it.

We did our version of all of it. The system under test does grounding: mapping a business metric such as revenue to the governed warehouse expression that actually computes it. Our benchmark is a board of 90 KPIs across four warehouses: our own demo SaaS warehouse, plus AdventureWorks, Chinook, and TPC-H. Every KPI has an audited oracle answer, and the whole board is frozen at a version number with a content hash, so a score today is comparable to a score last month. Changes ship behind flags, and a flag earns its way on by clearing a gate we write down before the run.

The eval system still lied to us four times.

Lie 1: "The score is real"

The first time we ran the full board honestly, this is what it said:

One note on denominators before we go on: the gate runs quoted later in this post score a slightly smaller groundable set than the launch board (our demo warehouse 24 of 25, AdventureWorks 22 of 25, TPC-H 19 of 20; Chinook is unchanged at 20). The frozen oracles are unchanged; the harness scores the rows in its groundable set, and every number in this post is pinned to a specific run artifact.

Zero. On TPC-H, the oldest and most standard analytics benchmark schema in existence, our grounding system could not correctly bind a single one of 20 KPIs to the warehouse.

To be precise about what these numbers are: internal development benchmarks on demo warehouses, not customer-reported accuracy. The failures were caught while building the system, before any of them could become a trusted business answer. That is the job.

Every instinct says to fix the board. Reword the KPIs, swap the warehouse, postpone the report. What we did instead was freeze it, hash it, and trace the zero. That trace was the most useful artifact of the quarter: 18 of the 20 misses came down to two infrastructure defects. Our relationship graph had only a single edge type on that warehouse, and our validator rejected Snowflake's numeric type before matching ever ran. The zero was not "the AI is bad at TPC-H." The zero was two named bugs, each with a file and a fix.

The second version of this lie is the one most demo environments are living in, including, at the time, ours. Our demo showed the same product at roughly 93% on our demo warehouse, because the demo map had been manually curated by hand. The automatic system underneath was at 28%. Both numbers were true. Only one of them measured the product. If you let the curated number stand in for the eval number in your own head, you will make roadmap decisions for a product that does not exist.

Where this stands now: the two TPC-H defects became ordinary bug fixes, and the warehouse was off zero within a week. The curated demo map is labeled a curated asset in our own docs, so nobody quotes it as a product number.

And since the vendor question at the end of this post cuts both ways, here is what the board did next, on the strict gate harness behind every number in the next two lies:

AdventureWorks moved because nine days of gated fixes landed. Chinook fell because a catalog rescan retired stale wins the linker had been leaning on, which means some of its earlier wins were never real. Across the four warehouses that is roughly 21% at launch to 34% by mid-July on the stricter instrument: three up, one down. Real improvement arrives lumpy and moves in both directions. That is how you know it is real.

A benchmark that cannot embarrass you cannot inform you. Version it, freeze the answers, and let it hurt.

Lie 2: "The bar was met"

Later that month we built a lever we had high hopes for: mining the warehouse's own query logs as evidence for which columns mean what. Before running anything, we pre-registered the success bars against the baseline of the day, July 15, which read 4 out of 22 correct bindings on the AdventureWorks section. The bars: at least 7 to turn the first flag on, at least 10 for the second.

Gate day arrived. Both bars were met.

They were met by the control run. On the same commit, with the feature switched off, the score was 10 out of 22. In the weeks between writing the bars and running the gate, fresh warehouse scans and unrelated merged fixes had moved the baseline from 4 to 10. Our feature, measured on versus off at the same commit, was byte-identical on every corpus. It moved nothing.

By the letter of the spec, we could have shipped it and reported a pass. The gate doc from that day calls this what it is: declaring a pass on drift the feature did not cause is bar-gaming in reverse. Both flags stayed off, and the machinery merged inert behind them.

The lesson generalized immediately: absolute bars rot. Catalogs get rescanned, baselines drift, other teams merge fixes, and that alone carried AdventureWorks from 4 to 10 while our feature did nothing. The only gate that survives contact with a moving codebase is a delta: the feature on versus the feature off, same commit, same data, written down before the run. This is also where I would push gently on the public playbooks. A fixed threshold like "clear 90% before rollout" is the right governance for launching a domain. For deciding whether an individual change earned its place, only the same-commit delta tells the truth.

Where this stands now: the query-log machinery is still merged and dark behind its flags, every gate since has run its off-control on the same commit, and absolute bars no longer decide anything.

Lie 3: "The eval says it failed"

For weeks, our board said TPC-H revenue KPIs were being grounded wrong, and the whole warehouse sat at 4 of its 19 groundable rows. The obvious move was to fix the product.

The product was right. The eval was wrong.

TPC-H revenue is a composite expression: SUM(l_extendedprice * (1 - l_discount)). Our production planner read the full expression and executed it correctly the entire time. But the eval's scorer compared expression strings, and the code path it used to extract the winning answer dropped the composite expression before comparing. Production kept the expression. The eval lost it. We were three days from rebuilding a working production path to satisfy a broken measurement.

It gets better. We prototyped the eval fix and dual-scored all four warehouses with it. Scored with the expression kept, TPC-H read 9 of 19 instead of 4; the entire difference was the revenue family being credited for answers it had been getting right. But the prototype also regressed our demo warehouse by one, on a metric where the system produced ABS(mrr_delta) against an oracle holding bare mrr_delta. Whether those are the same answer depends on the metric's sign convention: for a net change they differ, for a magnitude they agree. A string comparison cannot ask that question in either direction. There is no single right string for a metric. Sometimes the bare column is correct, sometimes the composite is. An eval that compares representations instead of meanings will punish both.

We dropped the fix rather than ship a scorer that punishes correct answers, and built the third path instead. Where this stands now: the harness carries an execution column beside the strict one. The headline verdict still compares expressions and stays strict, so it can never inflate. The second column executes the produced grounding against the live warehouse and credits equivalence when the numbers agree. Representation questions get settled in the only place they can be: by running the query. On the mid-July board, the strict column read 4 of 19 on TPC-H and the execution column confirmed three more rows whose queries returned the oracle's numbers, with one further row executing but ambiguous. The execution column is stingier than the string fix hoped, and that is the point: it credits nothing it cannot run.

Your eval is code. Code has bugs. When the eval and production disagree, the eval is not automatically the honest one, and the tiebreaker has to be replaying what production actually executed. We now treat a surprising eval failure the way we treat a surprising production failure: trace it to the row before believing it.

Lie 4: "It passed, so users have it"

This is the lie that produced two post-mortems in two days.

A grounding improvement passed its offline gate. The gate cells were green, the numbers moved, the branch merged. And in the live product, nothing changed. Not one row in the table that stores what users actually see was updated.

Two causes, both invisible to the gate. First, our eval harness never initialized the model-scoring component that production runs. In production, a reranker can demote a binding after the stage the harness measures. In the harness, that seam simply did not exist, so no gate cell could ever catch it. Second, the production write path had silent skip branches: conditions under which it would decline to update a row and say nothing. We eventually counted four distinct silent-skip defects on that one seam.

The uncomfortable summary from our own post-mortem: a green gate cell is an in-memory claim about the engine, not a claim about what persisted in a user's workspace. Those are different facts, and users live in the second one.

Where this stands now: the definition of done moved. A win is not realized when the gate passes. It is realized when the live database row shows the new binding, queried directly, in production. The write seam logs every skip with a reason, because a refusal that leaves no trace is indistinguishable from success. That check is part of closing out every grounding change we make.

The lie you cannot eval away

Suppose you dodge all four. Your board is versioned, your gates are deltas, your eval code is audited, your definition of done is a production row. There is a fifth problem, and it is the one nobody has solved.

Your eval measures your questions. Users ask theirs.

Anthropic's writeup is candid about the failure mode that remains even at 95%: the answer that is "wrong, but looks plausible and is used without objection." Silent failures do not appear in any offline score, because the person who would have flagged the error never noticed one. DABstep's hard tasks sit at 14.55% for the same underlying reason: the distance between a question as asked and the data as it exists is exactly the part that resists benchmarking.

The closest thing to a fix is a loop, not a benchmark. Uber's writeup of Cart Assistant, its grocery agent on Uber Eats, treats evaluation as a core part of the development loop rather than an afterthought: change the system, run the evals against baseline and candidate, and inspect regressions with step-level traces before deciding whether to fix the behavior or explicitly accept the trade-off. Their corpus mixes curated synthetic requests with anonymized production-derived edge cases, so production failures become permanent test cases. And sometimes the traces show that the evaluator, not the product behavior, was scoring the case wrong, which is Lie 3 at a company with far more users than we have.

We do not have a corpus of real user questions yet; we are early, and our board is built from the KPIs companies in each industry actually measure, with every oracle answer executed against the warehouse before it was frozen. So we treat our own top-line score with suspicion by policy. A 90% board would be a lab milestone, not a product guarantee, and we say so internally in those words. The pathway there is written down the same way the gates are: a ranked sequence of levers, each entering production only by beating a same-commit control. The one in flight right now is authority evidence: a definition mined from the warehouse's own transformation SQL should outrank a name-similarity score, because where a definition comes from should move how much it is trusted. What the levers point toward is not a bigger score. It is a system whose confidence is calibrated well enough that a stated answer can be trusted and a refusal means something. The product decision that follows from this is architectural: when the system cannot ground a metric with evidence, it refuses loudly instead of producing something plausible. A visible refusal can be counted, triaged, and fixed. A plausible fabrication becomes a number in someone's board deck.

And if you are buying one of these systems instead of building one, everything above inverts into questions for the vendor. Ask whether their gates are deltas against a same-commit control, or absolute bars that can drift into passing on their own. Ask what sits between an offline pass and the number reaching your screen, and how they would know if a write silently failed. Ask what the product does when its evidence runs out: refuse, or improvise. A vendor with a real eval system answers all three in specifics.

Build the eval that can hurt you

If you are building an AI data product, the AI data agent evaluation playbook that has survived contact with our production system looks like this:

• Version the benchmark and freeze the answers with a content hash. An eval you can quietly edit is an eval you will quietly edit.
• Pre-register gates as deltas against a same-commit control, never as absolute scores.
• Score the stages, not just the outcome. A final miss should be traceable to the component that caused it: candidate retrieval, semantic resolution, validation, execution, or the write that persists the answer. An aggregate accuracy number tells you that you failed. A component-level score tells you what to change.
• When the eval and production disagree, replay what production executed before trusting the verdict.
• Verify at the production row, not the green gate cell.
• Log every refusal and every skip with a reason.
• Harvest real corrections as eval candidates, the way Anthropic does. Your invented questions will drift from real ones.
• When the score rises smoothly, investigate. Real improvement arrives attached to specific named defects, and a smooth curve usually means the eval moved.

None of this is glamorous. Most of it is the kind of bookkeeping that gets skipped under deadline pressure, which is precisely why the skipped version fails. A serious eval is not a test you pass. It is an instrument that keeps telling you where the system is wrong. The eval system is not the scoreboard for the product. It is the part of the product that keeps the rest of it honest.

If you want to see the product these evals keep honest, take a look at how MindPalace works. For the system underneath, read about the data harness we built around a deterministic engine, or why we keep language models away from the math entirely. And if you are evaluating an AI data product rather than building one, ask the vendor one question: show me the benchmark run where your product scored zero, and what you did next. Teams with a real eval system have that story. Teams without one have a demo.