The outage begins at 9:41 on a Tuesday morning. You hear about it at 10:52, from a group chat.
The message is from an engineer you know at another company, and it is barely a sentence — anyone else seeing this? — over a link to the model vendor’s status page. The page has gone red. Not the amber of elevated latency; the red of a hard outage, the kind a vendor admits to only after its own support queue catches fire. The API under your assistant and half of your last two launch decks has been refusing the entire internet for more than an hour.
I should say up front that this morning is a composite, assembled from vendor outages I have watched teams shrug off and louder ones I have watched teams bleed through. Stay in it anyway, because which team you are is the most decidable thing in the story.
You pull your own dashboards and brace for carnage. Checkout is completing. Conversion is off by less than a point. The support assistant has stopped composing prose and is serving its plainer self: cached answers for the questions that repeat, a shorter path to a human for everything else. Onboarding opened accounts all morning and parked the personalization it could not perform — eleven hundred deferred jobs, waiting politely for the thinking to come back. The incident channel holds exactly one message, posted at 9:58 by the on-call: vendor down, flows on their fallbacks, banner up, nothing to pull until they recover. Nobody replied. Nobody opened a bridge.
Across town, a team with the same vendor and the same morning is on a bridge watching its checkout flow die, because that flow learned, sometime last year, to need a model for something a template used to do.
The difference between the shrug and the bleed was never the vendor, and it was never luck. It was decided months earlier, in unglamorous design reviews, by people who assumed the outage the way an architect assumes fire.
Every building you have ever worked in already solved this problem. The elevator is the most sophisticated machine in the building, and when it runs — which is nearly always — nobody takes the stairs above the third floor. The day it stops, a paper sign goes up and the building keeps working, because behind a heavy door sits the least impressive architecture in the place: poured concrete steps with no moving parts and no vendor. Nobody demos the stairs or puts them in a launch deck. The building code requires them anyway, on every floor, because the code was written by people who had watched buildings fail.
There is no such thing as a stairs outage.
The model is your elevator: better, and occasionally, unavoidably, absent. The only question that matters on that red morning is whether anyone poured the stairs.
Start with a label. Every model touchpoint in your product is one of two things. An enhancement makes the flow better when the model answers and leaves it standing when the model does not. A dependency is a capability the flow cannot finish without. That sounds like product philosophy. It is an operational fact, and the outage will assign the label for you if you decline to.
The test is mechanical: cut the call. For any model touchpoint, assume the call never returns — not slowly, never — and trace what the customer experiences. If they still reach the end of the flow, you have an enhancement. If they meet a spinner that outlives their patience, or a dead end in a friendly font, you have a dependency. Intent does not classify a touchpoint. Behavior does.
Run the test honestly and expect embarrassment. Most touchpoints turn out to be enhancements by intent and dependencies by implementation: the call site has no timeout, and the fallback branch was broken by a refactor months ago, because no test ever runs the flow with the model absent. An enhancement that can stall the flow is a dependency with better marketing.
The subtler finding is creep. Nobody decides to make checkout need a model. The dependency congeals: a template path deleted in a cleanup because the model handled everything for six straight months, a cache nobody warms anymore because fresh generation was always a little better. Renovations do this to buildings too. The stairwell fills with storage one box at a time, because the elevator has been so reliable — and the elevator’s reliability was never the point.
For the flows that pay salaries — checkout, signup, renewal, the support path that keeps customers from leaving — the fix has a blunt name: the zero-model-call path, a revenue-critical flow engineered to complete with no model calls. The force of the definition is its last clause. Not fewer calls. Not a smaller model. None. A floor with a model call in it is a floor that can be down, and the whole point of a floor is that it cannot.
The floor is poured from materials older than the model, and you own all of them already. Cached answers for the questions that repeat, and they repeat far more than intuition suggests. Templates from the part of last year your cleanup crews keep trying to delete. Rules, for the judgments that are lookups wearing judgment’s clothes. And deferral, the material teams forget and often the strongest: checkout’s contract is take the money and confirm it — not compose. The intelligence is genuinely valuable and genuinely not urgent, so accept the work now and park the thinking in a queue that drains when the vendor returns. Deferred is not degraded. The customer got everything they came for, and the polish arrived an hour late, which no customer in history has churned over.
When no material reaches — when the model really is the product — the floor is honest reduced service: a banner, written in peacetime, saying what is resting and what still works. Plain beats absent, and absent beats wrong. What customers remember is the bot that kept performing confidence an hour into an outage.
One more thing, because a fallback nobody has exercised is not a fallback. It is a rumor.
So unplug it. Pick a calm hour and turn the model off on purpose — fail the calls at the source, the way the vendor would. Staging first, until staging is boring, then production in a bounded window with the undo one flag away. The drill measures a single number: time to floor — minutes from the first failed call to the flow completing with no model call at all.
The first drill is a gift wrapped in embarrassments, which is what first drills are for. The touchpoint labeled enhancement that stalls the flow for ninety seconds. The template offering a product you discontinued in the spring. The banner nobody wrote. The switch that turns out to be a deploy. Every one of those was going to be discovered eventually. The drill just chooses the audience.
Expect the good surprise too. Teams that pour a first floor almost always find the work smaller than the fear: the templates still in version control, the rules the model replaced described, with no irony anywhere, in the prompt that replaced them. Days, not quarters. The fear was the reason it had never been scheduled.
Your model vendor’s next incident is already somewhere on the calendar. The first time your company takes the stairs should not be the day the elevator stops.