The most expensive meeting in your company has no numbers in it.
Picture the launch review. The AI assistant ships in two weeks, the roadmap slide has come and gone, and the one director who reads incident reports for fun asks the question the meeting was quietly built to avoid. How good is it?
Listen to what the room offers. The demos went well. Somebody from sales tried it on a personal account and loved it. Thumbs-up rate in the beta is strong, though nobody can say strong compared to what. Someone mentions that the new model is, and this is a direct quote, way better.
Every answer is a feeling doing a number’s job. And the meeting moves on, because the ship date is real and the question sounds soft. The question was never soft. It was just unmeasured.
I have watched enough versions of that meeting — a composite assembled from launch reviews across many teams, not one company’s story — to stop blaming the people in the room. They are not lazy. They are unarmed. Everything our profession perfected over twenty years detects absence: the crash that leaves a body, the timeout that leaves a timestamp. A model fails while fully present. The answer arrives on time and reads beautifully, and it is wrong — confidently wrong, no error code: a semantic failure. The service returns 200. The content is the failure. Nothing on the dashboard blinks.
Here is the part I need you to believe, because it is the whole game: you do not need new governance. Indicator, objective, budget, consequences — the SLO machinery you already trust survives contact with AI intact. What you need is one new competence underneath it: measuring the quality of judgment at scale. One team can stand up a working version in five days.
Start with a definition something can fail. “Gives accurate, helpful answers” is not a quality bar — no transcript could ever fail it, and no two readers will score it alike. A definition nothing can fail is a mission statement. The move, every time, is from vibe to verifiable. For a refund assistant, the working version is four plain sentences. States the refund window the policy table lists for the customer’s plan. Promises no credit before the eligibility check has run. Hands the conversation to a person the moment a denial is disputed. Answers the question the customer asked before saying anything else. Each sentence names where its truth lives. Each grades yes or no.
The doneness test is agreement, not elegance. Hand your sentences and twenty real transcripts to two people who did not write them, and have them grade alone. Where the scores match, you have a definition. Where they argue, you have found the next sentence to write.
Expect this work to surface a quiet war nobody knew was running. The policy page says fourteen days, the help center says thirty, and the assistant has been choosing daily. Settling which document is canonical sounds like eval bureaucracy. It is the first quality improvement you will ship, before a single test has run.
The definition needs something to grade, and that something is the golden set: a curated collection of real inputs with known-good properties, replayed against the system every time anything changes — a prompt edit, a model version. It is to judgment what your test suite is to code, and it earns the same respect. Nothing ships past a failing run.
The entries come from reality, which is the only author you should trust. Mine your transcripts for the questions customers actually asked, typos intact, because production will meet the typo and a cleaned-up version tests a customer who does not exist. Store expected properties, not expected answers — not the ideal reply’s text, but the sentence any good reply must satisfy. A scripted answer rots the first time the system words things differently, which is immediately. A property survives paraphrase.
Then adopt the rule that pays compounding interest forever: every wrong answer that reaches production becomes a golden entry the same week. The incident that embarrassed you in March becomes the test that protects you in April.
Measurement without consequence is where most quality efforts quietly die — not at the metric, at the teeth. So take your best indicator and give it what availability got: an objective and a budget. The assistant states policy correctly at least ninety-eight percent of the time, measured weekly — your number will differ, the shape will not. The two percent is a budget, and budgets exist to be governed. While it holds, teams ship freely. When it burns, a pre-signed policy fires: rollouts freeze and the canary path closes, and attention pivots from features to quality until the number recovers. Automatically. Without a meeting.
The load-bearing word is pre-signed. The first time a freeze blocks a launch someone important wanted, you will learn whether you built a policy or a suggestion — and that discovery goes better when the signature predates the launch by months and belongs to the executive now on the phone.
Here is the scene, stitched from budget breaches I have watched and freeze policies I have helped write. A staged rollout burns its weekly budget on a Wednesday. At 9 p.m. the policy does what it says: the rollout freezes, the morning promotion cancels itself, and a ticket arrives with the evidence attached. At 9:20 the product owner calls, and her arguments arrive in the order they always do — the number is barely over the line, the affected customers are a sliver, the demo is Friday. Every sentence is true. None of it is news. Each argument was made months earlier, in the peacetime negotiation that set the threshold, and the threshold is where they came to rest. You do not argue back. You send her the policy with her own signature on page two. The call changes subject inside a minute — not because anyone surrendered, but because she is arguing with her own signature, and losing.
The whole point of a budget is that nobody has to win a hallway fight at the moment of maximum pressure. The fight already happened, calmly, in a document.
So, the five days. Day one: pick the feature whose wrongness would hurt most and write what right means — a handful of checkable sentences, argued until the adjectives die. Day two: mine forty real inputs from your last ninety days of transcripts, thirty from the everyday middle and ten from the edges you fear. Do not invent inputs at the keyboard; that tests an engineer’s imagination, not a customer’s afternoon. Day three: wire the replay — a script and a folder, not a platform. It reads the forty entries and runs each through the real production path, writing one dated output apiece. Day four: turn the definition into binary checks and let a judge — a second model scoring outputs against your rubric, version-pinned like any instrument you calibrate — grade the run. Judges flatter fluency, so force every check to a yes or no and make the judge quote the line that decided its verdict. Then read every disagreement yourself, because each one is a defect in either the rubric’s wording or your definition of right. Day five: set the objective a notch below your measured baseline, and draft the one-paragraph breach policy.
The baseline will disappoint you. The deck said ninety-nine; the folder says ninety-one. Ninety-one is the first honest number the feature has ever had, and an objective set just under it will catch decline, which is its first job. Book the signature meeting while the number is fresh. Peacetime is a resource, and on Friday afternoon, holding a number that does not yet have enemies, you have more of it than you ever will again.
Five days. No platform purchase, no reorg. At the next launch review, when someone asks how good it is, you will not offer a feeling. You will slide a number across the table with a contract behind it.
The vibes meeting does not end because people stop having feelings. It ends the first time a number with consequences walks in.