The Missing Arkhipov

“…an insane and stupid gamble that nobody should be allowed to try.” Eliezer Yudkowsky and Nate Soares, on the attempt to build superintelligence, in If Anyone Builds It, Everyone Dies

This past weekend I read If Anyone Builds It, Everyone Dies, by Eliezer Yudkowsky and Nate Soares. It came recommended after I’d facilitated a couple of sessions on governance as strategy, built around Karen Hao’s Empire of AI. I went in expecting an adjacent argument. I came out somewhere else entirely.

I’ve read science fiction most of my life. Arthur C. Clarke, William Gibson, H.G. Wells, Aldous Huxley, and more recently Craig Alanson. What kept pulling me back to those books wasn’t the plot, it was the grounding. The chemistry, the physics, the engineering detail an author had clearly worked hard to get right before letting the imagination go further. That’s the art of the possible: earn the plausible mechanics first, then you’ve earned the right to speculate.

Clarke worked out how geostationary satellites could carry global communications in a 1945 paper for Wireless World, eighteen years before the first satellite reached that orbit; the orbit carries his name now. Wells coined the term “atomic bomb” in The World Set Free, written in 1913 and published in 1914, describing radioactive weapons dropped from aircraft three decades before Los Alamos, and Leó Szilárd read that novel in 1932 and later credited it with shaping what he understood a chain reaction would mean. Gibson invented the word “cyberspace” in a 1982 short story and sketched a global data network before most people had touched a modem. Huxley wrote about genetically stratified humans grown outside the womb and a pill that erases anxiety on demand in 1932. Alanson gave us an AI that’s smarter than everyone around it, runs logistics no human could keep up with, and treats its creators with open contempt, well before anyone had a name for what a frontier model’s personality might feel like. None of that was a lucky guess. It was writers who did the technical homework and then let the implications run further than most people were comfortable with.

This book gave me the same feeling, at first. Careful and grounded, willing to walk through the mechanism before it reaches a consequence. Then I looked into who Yudkowsky and Soares actually are: two people who’ve spent decades studying how smarter-than-human systems would think and what they’d want, who’ve put their names on public statements calling to stop this before it goes further. That’s when the feeling changed. Real shivers down the spine, not the fictional kind.

What unsettled me most wasn’t the extinction scenario itself. It was recognising the pieces already sitting on the table. A political leader willing to lie about what happened. State-linked hacking crews laundering stolen cryptocurrency to fund projects nobody should be funding. Convenient scapegoats produced on demand. None of that is speculative fiction. Some of it has already happened.

So here’s my honest recommendation: don’t read this over the summer break. Read it when you’re back at your desk, with the headspace to actually sit with it. If you love science fiction that’s grounded in real science, the kind that predicts rather than escapes, this will leave a mark. Because the authors are explicit that this isn’t fiction. It’s the future they consider likely if development carries on at its current pace, not one they invented for effect. I find the argument persuasive; plenty of serious people working on this do not, and I’d rather you read it and disagree with me than take my word for it.

The writers I grew up on didn’t get the future right by accident. They did the arithmetic and then refused to look away from where it pointed, whether that led to a uranium bomb, a data network, or a machine with its own agenda. Yudkowsky and Soares are doing the same thing. The uncomfortable part is that their arithmetic is aimed at something considerably bigger than a satellite orbit.

Two things came out of it that I’m still sitting with.

The first is narrow. I think we should stop developing humanoids. Not AI in general, not robotics in general. Specifically the decision to give a general-purpose intelligence a human-shaped body: hands, mobility, physical presence. A system that’s dangerous only inside a data centre is a contained problem. A system that’s dangerous and can also open a door is a different category of problem entirely, and we’re building toward it on purpose, as though the human shape were the interesting part rather than the risky one.

The second is bigger, and it’s the one that actually kept me up. On 27 October 1962, at the height of the Cuban Missile Crisis, the Soviet submarine B-59 came close to launching a nuclear torpedo at the American destroyer group above it. It didn’t happen because one officer aboard, Vasili Arkhipov, refused to give consent.

The detail that stays with me is why his refusal counted, and it’s worth slowing down on, because this is the whole point.

Four Soviet submarines were sent toward Cuba, each carrying one nuclear torpedo. The rule they were given was that if Moscow could not be reached, the captain and the political officer could authorise its use between them. Two names on the boat, two signatures, weapon released. On 27 October, out of contact for days, in a hull at over forty degrees, believing the war had already started, both of those men on B-59 wanted to fire.

The weapon stayed in the tube because of an organisational accident. Arkhipov was not part of B-59’s command. He was chief of staff of the whole four-boat flotilla, and he was travelling aboard B-59 that trip rather than one of the other three. Because a more senior officer was physically present, the authorisation rule on that boat had three names in it instead of two. He used the third name to say no.

Put it in governance terms and it gets uncomfortable. The veto that saved us was not a control. Nobody had designed a checkpoint into the launch procedure on the reasoning that a human should be able to refuse. What existed was a two-person rule, and two people said yes. The third seat appeared because of where one man’s desk happened to be that month. Had Arkhipov been assigned to B-4, B-36 or B-130, the rule on B-59 would have been the standard one and the standard one had already been satisfied.

That is the difference between a control and a coincidence. A control is triggered by the situation: it applies because a threshold was crossed, whoever is in the room. A coincidence applies because of who was in the room. They look identical in the moment they hold, and they behave completely differently the next time. We have spent sixty years telling the Arkhipov story as evidence that the system worked. It is better read as evidence that the system had a gap, and the gap happened to be filled by a man with the standing to object and the temperament to use it.

So I keep asking myself where the modern Arkhipov sits. I can’t find him, and the reason is worse than the nuclear case. There, at least, the seat was real even if the occupant was accidental: someone stood at a fixed point, a launch required a physical act, and refusing was a thing a person could do. In this one there is no seat at all. Not a designed one, not an accidental one. Thousands of data centres, dozens of jurisdictions, companies racing each other on a timeline nobody controls, and no point in the sequence where any named individual’s consent is required before the next capability is released. No hull to stand inside. No signature to withhold. The nuclear version worked by luck. This version has not even arranged for the luck.

It’s part of why the governance work I do keeps coming back to one rule: standards should only tighten, never loosen, once they’re set. That’s a small thing next to what this book is arguing for. But it’s the only lever I actually have.

---

Sources

The book

Eliezer Yudkowsky and Nate Soares, If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All, Little, Brown and Company, 16 September 2025. Published in the UK by The Bodley Head with the subtitle The Case Against Superintelligent AI. https://intelligence.org/2025/05/15/yudkowsky-and-soares-announce-major-new-book-if-anyone-builds-it-everyone-dies/

Karen Hao, Empire of AI: Dreams and Nightmares in Sam Altman’s OpenAI, Penguin Press, 2025.

Arthur C. Clarke

Arthur C. Clarke, “Extra-Terrestrial Relays: Can Rocket Stations Give World-wide Radio Coverage?“, Wireless World, October 1945, pages 305 to 308. Scanned original: https://lakdiva.org.lk/clarke/1945ww/

The first geostationary communications satellite, Syncom 2, was launched in 1963; Syncom 3 achieved a true geostationary orbit in 1964. The geostationary orbit is commonly referred to as the Clarke Orbit. Note that the concept of a geostationary orbit was described earlier by Herman Potočnik in 1929, a work Clarke cited; Clarke’s contribution was the communications relay application. https://www.historyofinformation.com/detail.php?id=2140

H.G. Wells and Leó Szilárd

H.G. Wells, The World Set Free, Macmillan, 1914 (written 1913). Wells’s bombs are made of a fictional element, Carolinum, and are dropped by hand from aircraft. https://en.wikipedia.org/wiki/The_World_Set_Free

Szilárd read the novel in 1932, conceived the neutron chain reaction in September 1933 while crossing Southampton Row at Russell Square in London, and filed a patent on it in 1934. He later wrote that he treated the patent cautiously because he had read Wells and understood what a chain reaction would mean. https://www.bbc.co.uk/news/magazine-33564130 and https://ahf.nuclearmuseum.org/ahf/history/early-atomic-science/

William Gibson

The word “cyberspace” first appeared in “Burning Chrome“, published in Omni, July 1982. It reached wide use through Neuromancer, Ace Books, 1984. https://en.wikipedia.org/wiki/Burning_Chrome

Aldous Huxley

Aldous Huxley, Brave New World, Chatto and Windus, 1932.

Craig Alanson

Craig Alanson, Expeditionary Force series, first volume Columbus Day, 2016.

Vasili Arkhipov and B-59

On 27 October 1962, the Foxtrot-class submarine B-59, commanded by Captain Valentin Savitsky and carrying one nuclear-armed torpedo, was forced toward the surface by depth charges from a US destroyer group accompanying the carrier USS Randolph. Out of contact with Moscow, Savitsky moved to arm the weapon. Boats in the flotilla normally required agreement between the captain and the political officer; B-59 required unanimity among three senior officers because Vasili Arkhipov, chief of staff of the flotilla, was aboard. Arkhipov withheld consent and the boat surfaced.

The episode became public in October 2002 at a conference in Havana marking the fortieth anniversary of the crisis, through declassified Soviet naval material presented by Thomas Blanton of the National Security Archive. Arkhipov had died in 1998. Details of the incident rest largely on later Russian participant testimony and remain debated among historians.

National Security Archive briefing book: https://nsarchive.gwu.edu/briefing-book/russia-programs/2022-10-03/soviet-submarines-nuclear-torpedoes-cuban-missile-crisis

State-linked cryptocurrency theft

The FBI attributed the February 2025 theft of approximately $1.5 billion in Ethereum from the exchange Bybit to North Korean actors it tracks as TraderTraitor, within the cluster commonly called the Lazarus Group. A UN Panel of Experts reported in March 2024 that North Korea was assessed to have stolen around $3 billion across 58 cyberattacks between 2017 and 2023, funds reported to support its weapons programmes. https://www.38north.org/2026/03/a-focus-on-cashing-out-one-way-to-combat-dprks-crypto-thefts/

---

On my use of AI

The thinking here is mine. The reading, the connections, the argument about humanoids, the Arkhipov parallel, and the position on standards that tighten and never loosen all came out of my own reflection, my own reading history, and the governance work I do with clients.

English is my second language. I use AI the way I would use a good editor: to structure an argument I have already formed, to tighten sentences, to catch the errors that a non-native writer makes and often cannot see, and to check facts, dates, and attributions against sources before anything is published. Where a claim in this piece is checkable, it has been checked, and the sources are listed above so you can check them yourself.

What I do not do is ask a model to have the idea. If a sentence here makes an argument, I made it. If it draws a line between a Soviet submarine in 1962 and a data centre in 2026, that line is mine.

I take full responsibility for everything published under my name, including any error that survived the process. If you find one, tell me and I will correct it in the open.