Four Failure Shapes From Public Postmortems

Most engineering writing about failure is written by the people who survived it. That is a limitation, but it is also the best data we have: a handful of teams have published detailed accounts of a system that did X, failed in a specific way, and was changed to Y — with timelines, numbers, and root causes attached. Read enough of them and the same four shapes start repeating, across domains that have nothing else in common: healthcare, e-commerce, journalism, market making, databases.

This article walks four of those shapes using public postmortems, orders, and peer-reviewed analyses, each cited so you can read the original rather than my summary. Several of the cases predate large language models by years, which is the point: the failure modes attributed to "AI systems" are mostly the failure modes of systems that act without a check, and that is an older problem than the models.

One thing I am not doing: presenting these as cautionary tales with clean morals. In every case below, the system was doing what it was built to do. The failure was in a check that was assumed — or merely asserted — rather than performed, and in almost every case the hard part was not fixing it — it was noticing.

Pattern 1: the label became the goal

This is the most consequential shape, and the least visible, because the model's accuracy metrics look fine the whole time.

The case: a health algorithm that predicted cost instead of illness. A widely used commercial algorithm allocated extra care to patients with complex needs by predicting something. The prediction target was not illness; it was health care cost. Because less money is spent on Black patients at equal levels of sickness, cost is a biased proxy for need, and the algorithm systematically under-referred them: at a given risk score, Black patients were considerably sicker than White patients. Obermeyer and colleagues, writing in Science in 2019, put a number on the correction — remedying the disparity would have raised the share of Black patients receiving additional help from 17.7 percent to 46.5 percent — and drew the general lesson in one sentence: choosing convenient, seemingly effective proxies for ground truth is itself a source of algorithmic bias.

The "Y" here is not a better model. The fix was to change the label: predict health outcomes rather than cost, using the same class of model on the same data. The bias was in the objective, and no amount of model improvement removes a bias in the objective.

The case: an AI recruiting tool that learned from hiring history. Amazon built machine-learning resume screening from 2014 which, as Reuters reported in 2018, "penalized resumes that included the word 'women's'" and downgraded graduates of two all-women's colleges. The training signal was a decade of hiring decisions, which encoded the past rather than the intent. Amazon scrapped the tool; the reported conclusion inside the company was that it could not be made to work reliably.

The common structure: a proxy label is chosen because it is measurable, cheap, and correlates with the goal; the model learns the proxy faithfully; the divergence between proxy and goal is invisible in the loss curve and only shows up as an outcome someone investigates by hand. Note what accuracy would have said here — it would have said the model was good at predicting cost, and good at predicting who had been hired. Both true, both useless.

Pattern 2: the output shipped without an effective gate

Shape: a generative output reaches the audience — or becomes policy — with no effective check between the model and the consequence. In one of the cases below a check was claimed in writing, on every article. It did not hold.

The case: 41 corrections out of 77 articles. From November 2022, CNET published financial explainers written by an in-house AI tool under the byline "CNET Money Staff". The disclosure existed, but only in a dropdown, and it read that the article "was generated using automation technology… and thoroughly edited and fact-checked by an editor on our editorial staff". No announcement was made. As far as Futurism's reporting in January 2023 could establish, the first person to notice was not an editor, a monitor, or a journalist — it was an online marketer posting about it. The story became a story only after that. CNET then ran an internal review and issued corrections on 41 of the 77 articles, paused the program, and put the content through editorial review. So the failure was not that a model wrote prose, and not that a gate was missing — a gate was asserted on the face of every article. It is that an asserted gate is not a gate: the check existed on paper and did not hold at the rate that 41 corrections implies, and the first detector was a stranger on social media rather than anyone inside the building.

The case: the chatbot whose words became the airline's words. In Moffatt v. Air Canada (2024 BCCRT 149), a passenger asked the airline's website chatbot about bereavement fares and was told he could apply after booking, contrary to the actual policy. The British Columbia Civil Resolution Tribunal found Air Canada liable for negligent misrepresentation and ordered it to pay the fare difference plus interest and fees. The airline's position that the chatbot was a separate entity responsible for its own statements did not hold; the chatbot's answer was the company's answer.

The two cases differ in what was missing. CNET had a named check and it did not hold; Air Canada had a statement about policy, and no verifier between the chatbot and the customer. What they share is the boundary: something that is not the model has to decide whether the output crosses it, and that decision has to be real rather than asserted. The Air Canada case adds the accountability dimension the CNET case does not: once the output is a statement about policy or price, publishing it unverified is not a quality problem, it is a liability.

Pattern 3: validated inside, deployed outside

Shape: the model performs acceptably in the environment where it was built, and the rollout decision is made on that basis. Performance in the deploying environment is treated as a continuation of the same number.

The case: a sepsis alert running in hundreds of hospitals. The Epic Sepsis Model is a proprietary prediction tool inside a widely deployed electronic health record, implemented at hundreds of US hospitals to alert clinicians that a patient may be developing sepsis. Wong and colleagues published an external validation in JAMA Internal Medicine in 2021 and found poor discrimination and calibration. Their conclusion is the sentence worth keeping: the widespread adoption of the model despite its poor performance raised "fundamental concerns about sepsis management on a national level".

What makes this the sharpest of the four shapes is the asymmetry of the failure. A model that is wrong in a demo is a demo that did not work. A model that is wrong at national scale, behind an alert, is a system that spends clinicians' attention — and alert fatigue is not a neutral outcome, it changes how people respond to the next alert.

The mechanism is worth being precise about, because it explains why this failure is invisible rather than careless. A model trained on one population's data learns that population's base rate. Run the same model where the condition is less common and the score distribution shifts under a fixed cutoff, so the same threshold produces a different ratio of true to false alerts. Discrimination — can it rank the sicker patient higher — travels better across populations than calibration, which is whether a given score means the same probability here as there. An alert is a threshold decision, so it depends on both. That is how a model ends up defensible on the data it was built from and harmful on the data it meets, with nobody having done anything obviously wrong.

The Y in this pattern is procedural, and it is the least glamorous work in the whole article: validate in the environment you are deploying to, on that population's data, before rollout — and treat a vendor's internal numbers as a claim, not a result. The authors argued exactly that, after the fact.

Pattern 4: the deploy, and the recovery, were never tested

The last shape has nothing to do with models, which is why it belongs here. Both cases are about automation that was trusted in the one path that had never been exercised.

The case: seven servers out of eight. On 1 August 2012, Knight Capital deployed new code to eight servers. One did not get it. On that server, repurposed feature flags activated dormant code, and when the market opened the firm accumulated an unintended multi-billion dollar portfolio in roughly forty-five minutes. The US Securities and Exchange Commission's order put the loss at more than $460 million, noted the absence of any procedure requiring a second person to review deployments, and imposed a $12 million penalty. There was no kill switch, because nobody had needed one yet.

The case: four ways to restore, one of which worked. GitLab's own postmortem of the 31 January 2017 outage is the most useful document in this article, because it lists what was believed rather than what was true. The incident began with an engineer running a destructive command on the primary database instead of the secondary; around 300 GB was removed. What turned an accident into an 18-hour outage plus permanent data loss was the recovery layer. GitLab had four mechanisms on paper — daily pg_dump backups to S3, daily LVM snapshots used to refresh staging, Azure disk snapshots for various servers, and streaming replication to a standby. In the event: the pg_dump backups did not exist because the tool was a mismatched major version and had been erroring for some time, and the failure notifications went by email that was rejected because DMARC was not configured, so the failure was silent; the Azure disk snapshots were never enabled for the database servers, on the assumption that the other procedures were sufficient; the LVM snapshots were not intended for disaster recovery; and replication was broken as part of the incident, with the standby wiped during a failed rebuild. The restore ran off a six-hour-old snapshot and took roughly 18 hours. Around 5,000 projects, 5,000 comments, and 700 user accounts were lost.

GitLab published the root-cause analysis and fifteen tracked remediation items, including automated testing of backup recovery and, explicitly, assigning an owner for data durability — the postmortem's own diagnosis of why a broken backup had gone unnoticed was "there was no ownership". The line I would put on a wall is theirs: an ideal environment is one where you can make mistakes and recover quickly, which requires that the recovery procedures are exercised routinely rather than assumed.

What the four shapes share

Read together, the cases collapse into one observation. In none of them was the primary component broken. The cost model predicted cost. The resume model learned the hiring history. The generative tool produced fluent prose. The deployment tool deployed to seven of eight servers, exactly as instructed by a human. The backup mechanisms were individually reasonable designs.

What was missing, in every case, was the same thing in different forms: a check on a boundary that everyone assumed was covered, and that could demonstrate it was working. CNET had the check on paper and no evidence it held; the other cases did not have one at all.

• The objective boundary. Is the label we are optimizing actually the goal, or a convenient proxy for it? (Cost versus illness. Hiring history versus hiring intent.)
• The publication boundary. What stands between the generated output and the audience, and does that check hold? A named check is the start: an editorial pass, a verifier, a human who signs. The evidence that it works is a rejection — a recent real one, or, where the failure is too rare to arrive on its own, a drill that forces it. CNET's gate was named on every article and let 41 of 77 through.
• The environment boundary. Has this been validated where it will run, or only where it was built?
• The recovery boundary. Has the restore been executed, recently, by someone who is not the person who wrote it?

And one second-order pattern, which is the one I find most transferable: detection was the hard part, not prevention. The biased algorithm looked accurate. The CNET errors surfaced because a stranger on social media noticed, not because anything inside CNET was watching. The GitLab backup had been failing silently for long enough that nobody remembered it working. The Knight deployment looked complete. In each case the system had no signal that distinguished "working" from "confidently wrong", which is the same problem this blog keeps returning to: a failure that does not announce itself needs an instrument that announces it, and building that instrument is what the companion piece on observability for AI agents is about.

What these postmortems cannot tell you

Four caveats, because the temptation to over-read this genre is strong.

Survivorship bias, twice over. You are reading about teams that survived the incident, had the resources to investigate it, and chose to publish. For every Knight Capital there are unnamed firms whose deployments failed the same way and whose story is a private settlement. The published set is not a sample; it is a selection of the best-documented cases.

Narrative smoothing. Postmortems are written after the fact by people who now understand the cause. Real incidents are not experienced as a tidy causal chain; they are experienced as confusing symptoms and competing hypotheses. The "five whys" in the GitLab document is a reconstruction, and reconstructions make the diagnosis look more inevitable than it was.

The proposed fix is often "we stopped". Amazon scrapped the tool; CNET paused the program; Air Canada's chatbot was disabled. Stopping is a legitimate fix and a poor template. It tells you that the approach was not salvageable in that context, not what to do in yours.

Your blast radius is not theirs. A recommendation model that is wrong costs a click. A sepsis alert that is wrong consumes clinical attention at national scale. The same technical failure has wildly different acceptable frequencies, so the transferable content of a postmortem is the shape of the failure, not its severity.

How to use someone else's outage

The practical value is not in remembering the cases; it is in having the four shapes available at the moment you design something, so they act as questions rather than as reading material.

Concretely, before shipping anything where a model's output reaches a user, a system of record, or a clinical or financial decision, walk the four boundaries and write down the answers. Not the intentions — the answers, with names attached.

That gate block is the same family of questions I use when deciding whether a model should be allowed to act at all: reversibility and detectability are what make a failure cheap or expensive, and they are the two properties the boundary essay keeps coming back to.

Two lines in that file are doing more work than the rest, and both come straight out of the cases. The first is silent_failure_check: GitLab's backup did not fail because someone chose a bad design, it failed because the notification path was broken and nobody owned the mechanism. Any monitoring, backup, or verification you rely on should have an explicit answer to "how would we know it stopped working", and the answer should not be an email that might be rejected.

The second is detectability. Several of these incidents were cheap to fix and expensive to notice. Asking ahead of time what signal would distinguish a working system from a confidently wrong one is the difference between an incident measured in hours and one measured in months — and it is the question that none of the postmortems above asked until after the fact.

There is an obvious objection to the rejection test, and it is worth answering rather than discovering in the comments: for a control whose failure is rare and catastrophic, never rejecting anything is the desired state. A backup restore that has never been needed is a backup restore that has never failed, and reading that as absent evidence would be wrong. The resolution is that you stop waiting for nature to produce the occasion and manufacture it on a schedule — a restore drill, a chaos experiment, a red-team run, a form filled in with a name and a date. An untested kill switch and a kill switch that has fired in a drill are different artefacts, even though neither has ever fired in production. GitLab's remediation list makes the same move: among the fifteen issues is automated testing of database restore, which is the drill, sitting next to monitoring for the backups, which is the part that tells you the drill stopped running.

And one objection to the file itself, since it is the failure this article is about one level up. A checklist in a repository becomes a paper process unless something enforces it: the file does not gate a deploy, raise an alert, or stop a launch. If the answers get filled in and nothing else changes, you have built a gate that is asserted rather than performed — the CNET case, at lower stakes. The minimum is to bind each line to something that fires on its own: a CI job that fails, a monitor that pages, a launch checklist with a name against every item. Written down is the beginning of the control, not the control.

Summary

Four failure shapes, drawn from seven published incidents. An algorithm that predicted cost instead of illness, and under-referred Black patients while its accuracy looked fine; a hiring model that learned a decade of biased decisions and was scrapped; 41 corrections out of 77 AI-written articles; a chatbot whose invented policy became the airline's liability; a sepsis model adopted at hundreds of hospitals before anyone validated it externally; a configuration that reached seven of eight servers and lost more than $460 million in forty-five minutes; and four recovery mechanisms of which only one proved usable.

None of those systems was broken. Each of them was missing a check that actually held, on a boundary that was assumed — or merely asserted — to be covered: the objective, the publication, the environment, the recovery. And in almost every case the expensive part was noticing, not fixing.

That is the transferable content. Not "AI is risky", and not the specific remediations, which were all context-dependent. It is that a system which can act, and cannot tell you it is wrong, will eventually act wrongly for longer than you can afford. The postmortems are worth reading for the shapes; the work is naming the checks in advance — and being able to show that each one has said no at least once, whether the occasion came from production or from a drill you scheduled.