Why a human has to approve every metric

The seductive version of natural-language BI is three lines long: hand the model the schema, hand it the question, run whatever SQL comes back. It demos beautifully. We built the other thing, and this is why.

The failure mode is not what people expect

The worry people bring to text-to-SQL is that the model will produce SQL that does not run. That worry is misplaced — broken SQL is safe. It throws, someone sees the error, nobody makes a decision on it.

The dangerous output is SQL that runs, returns a number, and is quietly wrong. Consider a column named AMT on a trades table. Is it notional? Settled value? Gross or net of fees? In which currency, on which date’s rate? A model will pick one reading, and the reading will be plausible. Plausible and wrong is the worst possible property for a number that is about to be pasted into a board deck.

Nothing about that failure is fixed by a better model. It is not a reasoning failure. The information required to answer correctly is not in the schema — it lives in the heads of the four people who have worked on that warehouse for a decade.

So we moved the guessing offline

Insighter splits into two stages that never blur.

The knowledge build runs offline, before any user question exists. It walks the warehouse and proposes a catalogue: what each table represents, which columns carry business meaning, how tables join, and what each metric computes. Everything it produces is a draft.

A person reviews the drafts. They approve, correct, or reject them. Only approved entries are published into the catalogue.

Runtime may only use the published catalogue. A question is resolved against approved business terms, planned against approved joins, and turned into SQL using approved metric definitions. The model has enormous freedom in how it interprets a question and no freedom at all in what the words mean.

What this actually buys

Three things, and the third is the one that matters commercially.

1. The wrong-number class of bug largely disappears. Not because the model got better, but because the ambiguity it would have guessed at was resolved once, by someone who knew.

2. Answers are checkable. Every answer ships with the SQL and the catalogue entries behind it. A sceptical analyst can verify it in under a minute, which is the only speed at which people actually verify things.

3. Someone can sign off on it. In a regulated business, “the AI decided that notional exposure means this” is not an answer. “Your head of risk approved this definition on this date” is. The catalogue is the audit trail.

What it costs

It is not free, and pretending otherwise would be dishonest.

There is real human work up front — someone has to sit with the proposals for a few hours per warehouse. Coverage grows over time rather than arriving complete, so early on there are questions the system will decline to answer instead of guessing. That declining feels worse than a confident answer, right up until the first time the confident answer would have been wrong.

And schemas move. That one we solved in code: republishing a catalogue is a merge, not a replacement. Approvals already given survive the next build, or the review burden would compound forever and the system would be abandoned within two quarters.

The general principle

The trade generalises past analytics. An agent earns autonomy in proportion to how cheaply a human can check it. Every agent in the suite is built to the same shape: bounded by something a person approved, and transparent about the working it did inside those bounds.

Speed you cannot verify is not speed. It is just risk arriving faster.