Maybe it was worse to be right about AI
Dylan Matthews at Vox recently lamented how he’d gotten it so very wrong about AI.
He’d gone to an Effective Altruism conference (EA Global SF) in 2015, so very early-days EA (I came to the movement in 2013 or so, and took the Giving What We Can pledge in 2016). There was lots of talk of how an artificial intelligence could cause existential catastrophe if not aligned with human interests; Bostrom’s Superintelligence had come out the previous year; AGI was in the water.
Though there had been exciting advances in deep learning at the time (AlphaGo, using deep RL to play Atari and the like), the idea of a superintelligent machine taking over felt fanciful. Even more fanciful was the idea that we should commit significant resources to mitigating the risks from such a machine. Comparisons to Terminator abounded (and people rolling their eyes at same).
Then the last five years happened.
Matthews definitely got it wrong – on the object level, but also, he admits, on the meta level: he didn’t take the idea seriously because it was weird.
But sometimes being right is worse. Scientists interested in viruses were right that they could be extremely dangerous, but doing gain-of-function lab research on them plausibly caused the COVID-19 pandemic. Likewise, many EAs who were concerned about AI ended up in key roles at companies accelerating the development of technology they feared.
Creating the Torment Nexus
Paul Christiano and Dario Amodei, both early EAs, co-invented RLHF (reinforcement learning from human feedback) as an alignment technique. This is the post-training step by which LLMs are trained to be helpful assistants. Without RLHF, they just parrot back plausible words from their training corpus, like early GPT models. This turned out to be essential for building the dangerous thing.
A small cadre of EAs figured out scaling laws early, realising that LLMs could be scaled enormously with highly predictable increases in performance. This derisked pouring hundreds of millions of dollars into training runs; without knowing how the models scale there would have been no appetite for this. Also essential for building the dangerous thing.
The counterfactual is unknown, admittedly – perhaps all this would have happened anyway. But by Karen Hao’s account, Dario Amodei (as an example) was pushing hard for scaling while at OpenAI, even while others were skeptical. He viewed scaling language models as the fastest path to AGI, and that OpenAI should devote enormous amounts of compute to the task.
OpenAl had already scaled up GPT-2 into something modestly larger. But Amodei wasn’t interested in a modest expansion. … Microsoft was about to deliver a new supercomputer to OpenAI as part of its investment, with ten thousand Nvidia V100s … Amodei wanted to use all of [them] to create the new large language model. …
Many OpenAI researchers were skeptical that Amodei’s idea would even work. Some also argued that a more gradual scaling approach would be more measured, scientific, and predictable. But Amodei was adamant about his proposal and had the backing of other leaders.
He subsequently quit OpenAI to found “AI safety lab” Anthropic, which too went on to create frontier AI models. Anthropic actually released the first version of Claude as a private research preview before the release of ChatGPT.
It feels similar to Leo Szilard’s realisation that a nuclear fission chain reaction could be used to create a bomb. Initially he deliberately withheld his findings from the broader scientific community. He filed a patent application the year after his discovery, and in the following years urged his colleagues not to disseminate the ideas further. It was only when it became apparent the Germans were building the bomb did he switch tack and push the US government to develop nuclear weapons of their own.
Except that in this case, it’s not obvious that anyone was pouring millions of dollars into LLM training runs (or indeed had any desire to). Perhaps we should’ve started with Szilard’s first plan instead of going straight to the second.
The Kimi moment
Spurred on by American efforts, a Chinese lab has now released an open-source model which is by all accounts nearly as good as Anthropic’s Opus 4.8 at a fraction of the price. We decided to go hell for leather on scaling LLMs so we could be ahead of the Chinese, but it seems just as likely that in doing so we just triggered a race to the finish. (What ‘the finish’ looks like is left as an exercise to the reader.)
In a sort of hilarious irony, Chinese models are now able to perform cyberattacks which US models refuse to mitigate. Huggingface was this week the victim of an AI agent-driven cyberattack (either by an open-source model or a hosted and jailbroken one), but found US frontier models were unwilling to help because of their guardrails:
When we started the log analysis, we first used frontier models behind commercial APIs. This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers’ safety guardrails, which cannot distinguish an incident responder from an attacker. We ran the forensic analysis instead on GLM 5.2, an open-weight model, on our own infrastructure.
There’s no way back. Now that we are racing to build and deploy superintelligent AI, it’s probably better that the US has the edge rather than China. Better still would be a multilateral agreement to slow or pause frontier development, as the leaders of basically every frontier lab are now calling for (most recently Demis Hassabis11. Although his was a pretty milquetoast proposal for a FINRA-like regulator in the US to vet models which frontier labs optionally submit for assessment. Much too little, much too late!). But one has to wonder whether being right about AI led the EA movement down a path to helping build the very thing it was trying to control.