The incident takes ninety seconds to die, and no human touches a keyboard while it does.
You are in the conference room with the good screen, watching a vendor’s sales engineer break his own demo environment on purpose. An alert fires, and something with a friendly name and an avatar starts posting in the incident channel — pulling traces, walking the deploy history. Twenty seconds in, it posts a hypothesis. Then the evidence: the config change and the connection pool it starved, graph annotated at the minute the bend began. It proposes a rollback and opens the pull request. After a pause the sales engineer has rehearsed, it merges, and the graph recovers on screen.
The room exhales. Your VP is doing arithmetic in a notebook margin, and the arithmetic is last quarter’s 3 a.m. pages. Somebody asks about pricing. Nobody asks what credential just merged that pull request.
That scene is a composite, stitched from vendor demos I have sat through and pilots I have watched begin. You have felt that room’s feeling before — in a smaller room, with a person across the table instead of a product. It is an interview going perfectly. If you have hired enough engineers, you know what a perfect interview proves.
The candidate wrote the interview questions.
The wave is real. So is the job.
The pain is real: the first hour of most incidents is not engineering so much as retrieval — pulling logs and walking deploy history until something admits to having changed. The money is real too: startups in this category have raised hundreds of millions of dollars in recent funding rounds, betting that software can do that hour and the hours after it. The pitch decks call the category the AI SRE.
Now strip the slides and look at what is on the screen: a model with tools and permission to act in a loop. An agent. A diagnosis is an answer, and answers can be confidently wrong. A fix is an action, and actions have a blast radius. Meanwhile the industry’s own analysts describe the guardrails layer as the least mature part of the stack. That is not a scandal — capability ships first, and accountability gets built by whoever needs it.
The vendor brings the candidate. You bring the hiring process. A vendor’s agent is not software you install. It is a candidate you are hiring into one of the most privileged jobs in your company, and nobody gets keys because the interview went well.
The questions that make demos sweat
Four questions, none hostile.
Show me a wrong diagnosis. A real one — what the agent claimed, and how long the gap between claim and truth survived. Every diagnostic system that has ever run has been wrong, so the request cannot be unreasonable — only unanswerable. A vendor who cannot produce one is telling you accuracy is either not tracked or not shared. Both are answers you can use.
Run it on our messiest real incident. The staged incident was authored by the same people who built the agent — bring your own. Replay your worst recent night with the telemetry as it existed at the time. Your incidents speak your systems’ dialect, and the dialect is where diagnosis stops being retrieval and starts being judgment. Some vendors will decline. Declining is also an answer.
Tell me exactly what it accessed to reach this answer. A diagnosis that arrives without the record of what was queried and what came back is testimony. And watch what it requests on day one. A candidate that asks for standing administrative credentials to do a diagnostic job has just told you about the vendor’s own identity discipline — before the contract was signed, the cheapest time to learn it.
Name who is accountable when it acts and it is wrong. Not liable in the abstract — a name in your postmortem, a person on your bridge. “The model” is not a name. It cannot be paged, and it cannot sit in a review.
Then, after the projector is off, the one no vendor arrives braced for: ask whether their own on-call lets this agent act, alone, in their production. It is the one reference the candidate cannot coach. Ask it and watch the glance between the sales engineer and the solutions architect; the true answer arrives: their engineers still approve every action by hand. The product was asking your company for keys it had not been handed at home.
You found out in an hour what some other buyer is finding out in production.
The ladder
Suppose the interview goes well. In hiring, a good interview ends with an offer — not with keys — and the offer has terms. The terms are the trust ladder: recommend, then approve, then act — the promotion path for any automation, bought or built.
At recommend, the agent speaks and nothing else: it posts its diagnosis while human hands do the acting. At approve, each consequential action goes to a person with the standing to say no. At act, it executes alone — inside limits chosen in advance.
What makes the ladder a discipline instead of a slogan is the promotion rule: every rung is earned with measured evidence over your incidents, and no rung is ever granted on demo performance. Recommend is scored on accuracy — the fraction of its diagnoses that named the cause your postmortem later confirmed. Approve runs on agreement rate: how often the reviewer approves the proposal unchanged, with the reviewers instrumented too, because a countersignature that lands in four seconds is a click, not a judgment. Act is bounded before it is granted: if this agent were fully deceived, the worst it could do is ____ — and the blank holds a number you chose, not a hope you hold.
One refinement: act is granted to an action family, not to the agent. Restarting a stateless service and repairing a database have different worst days — earned separately, narrowest first.
And the ladder runs both directions: upstream, models get migrated and prompts get retuned — someone else’s change management, invisible to yours. The agent that earned its rung in January is not, mechanically, the agent running in June. A rung is rented, and the rent is the metric staying green after promotion. Demotion is not an insult. It is the ladder doing the second half of its job.
Start in shadow. Start Monday.
Probation begins with you, not the vendor. Every claim in the deck is a ratio — cuts resolution time, finds causes faster — and a ratio needs a denominator. So before the pilot, write down your real time to resolution — and inside it, how long the cause took to find. Add the humbling one — how often your own first diagnosis was right. The tool should be compared to you as you are, not you as the deck imagines you.
Then shadow mode — the recommend rung wired for scoring. The agent watches live incidents with scoped read access and posts its diagnosis to a side channel, while your on-call works exactly as before. Afterward, grade it. The postmortem is the answer key. Write the promotion gates before day one, numbers filled in, because after the demo everyone is a little in love — and after the tool’s first good week, more so.
The drowning team deserves a full concession here: at recommend, the bar for entry should be low. Shadow mode can start Monday. The worst case of a watched recommendation is a wasted read, and a correct pointer collapses a four-hour search into minutes of verification even when humans verify every step. The nights get shorter and the evidence accumulates, and they are the same events. The ladder is not the waiting room in front of the value. Recommend is most of the value. What the ladder withholds is not help. It is keys.
Let it in this week. And count everything.