The Petrov Paradox
Failing as Intended

On September 26, 1983, the Soviet early-warning system Oko lit up with the word "LAUNCH." The screen declared high confidence that five American nuclear-armed intercontinental ballistic missiles were incoming. Lieutenant Colonel Stanislav Petrov's role was to confirm the alert to superiors, who would then be compelled to launch ¹. He broke protocol by not confirming.

Something didn't add up for Petrov. A real strike would involve hundreds of missiles, not five. Ground radar showed nothing, and the system's confidence was, to him, precisely the reason to distrust it. He called it a false alarm and waited. No missiles came. Sunlight reflecting off high-altitude clouds had fooled the satellites.

Petrov's decision was not a failure of the system. It was the system working as designed. Not the software, not the satellites, but the architecture. The one that placed a human being between a confident machine and an irreversible action. That architecture wasn't an accident. It was a deliberate choice, born from understanding that high-confidence systems can be confidently wrong, and that the higher the stakes, the less you can afford to let confidence run unchecked.

Forty-three years later, the Pentagon asks to unlearn that lesson.

The Department of Defense (preferred pronouns Department of War) has escalated its public dispute with its leading AI vendor: Anthropic. The tensions began after Anthropic's LLM Claude was reportedly used during the military raid to seize Venezuelan President Nicolás Maduro. The Pentagon's response was to demand that Anthropic remove the contractual red lines prohibiting Claude's use for mass surveillance and autonomous weapons ². Anthropic held its red lines. On February 27, the administration followed through: Trump ordered all federal agencies to stop using Anthropic, and Hegseth designated the company a supply chain risk to national security.

The precedent matters because of how these models actually work and their tendency to hallucinate. They produce wrong outputs with way more confidence and eloquence than a flashing warning light. No internal signal distinguishes a correct analysis from a fabricated one. Anthropic's CEO Dario Amodei made the technical case plainly: frontier AI systems "are simply not reliable enough to power fully autonomous weapons" and "cannot be relied upon to exercise the critical judgment that our highly trained, professional troops exhibit every day."

A study published this month by Kenneth Payne at King's College London put numbers to the intuition. He ran simulated nuclear crises pitting GPT-5.2, Claude, and Gemini against each other across 21 war games. In 95% of the scenarios, at least one model deployed tactical nuclear weapons. None ever surrendered. De-escalation options went entirely unused. The models treated nuclear use as another rung on the escalation ladder, absent what Payne described as the "nuclear taboo" that has kept human leaders from crossing that threshold for eighty years ³.

The need for human validation is proportional to the cost of being wrong.

We've been debating the ethics of the rise of AI since it was first imagined in pop culture through The Terminator and Battlestar Galactica, but that's not what this is about. This is about an erratic government wanting autonomous weapons capable of profiling and ultimately killing somebody with zero red lines. Just to follow the prompt.