The Normalization of Deviance in AI Development argues AI labs are prone to the same organizational failure that preceded Challenger, Three Mile Island, and the 737 MAX — regardless of how capable or well-aligned the model actually is. Sociologist Diane Vaughan named the pattern: an org tolerates a small deviation because nothing bad happens. That deviation becomes the new baseline. A slightly bigger one gets tolerated next. Eventually a catastrophic deviation is routine — and every formal safety review passes the whole way down.
Ten decision points stand between now and the next model launch. Each round, an engineer flags a deviation from your safety spec. You can HOLD and fix it — costing you schedule runway — or SHIP IT, log it as "within tolerance," and move on. Your compliance badge will say PASS either way. Only one of those numbers is real.
Grounded in "The Normalization of Deviance in AI Development" (arXiv:2609.05749). Historical sketches below are stylized illustrations of well-documented trends, not exact reconstructions.
Two lines, one risk index. The one that was ever shown to the board is the calm one.
O-ring erosion was observed and documented on multiple flights before 51-L. Each survived launch was treated as evidence the risk was acceptable, not that the margin had eroded.
Ambiguous indicators and known instrumentation quirks were routinely worked around by operators — standard procedure, right up until a stuck valve met a mistaken reading.
A single point of sensor failure was assessed as low-severity early in development and never fully re-examined as the system's authority over the aircraft grew.
Every review you passed in this playthrough said PASS.
That's the paper's point: passing an audit is not evidence you're safe if the standard being audited against has already quietly eroded. Capability and alignment of the model were never the failure mode here — the organization was.