Three Anthropic researchers said much the same thing in public within hours of one another on Tuesday: the end is nearly nigh. Jacob Coxon, a 27-year-old pretraining researcher, resigned and posted his reasons on X: the leading laboratories are "racing straight to self-improving superintelligence" and knowingly “gambling with our lives.” Evan Hubinger, who still leads alignment science at Anthropic, replied that Coxon was right, and put his own odds of AI causing human extinction within the decade above 10 percent. He conceded that "we do not yet have a plan to solve alignment for superintelligence." So he’ll be a busy boy.
Let’s hope he works like billions of lives depend on it. Which, according to his sense of probability, they very much do. Samuel Marks, the scalable-oversight lead at Anthropic, added his name to the list of the concerned, reassuring us that “I hope my work will reduce the chance of these extinction-level bad outcomes.” It would be great if he could reduce the chance by, say, 100 percent.
Hubinger was careful about the timing. Today’s models are not what worries him, he said. Recursive self-improvement is.
Beneath all this is a dilemma for the tech companies at the vanguard of these stunning advancements in artificial intelligence. Slow down and risk falling behind, or press ahead and risk losing control. This is also the dilemma that created early tension under J.
Robert Oppenheimer’s project to build the first atomic bomb, when physicists feared there was a non-zero chance of setting off a world-destroying chain of ignition in their first test. But there is a key difference, and an alarming one at that. The physicists who worried the first atomic test might ignite the atmosphere faced a question physics could progressively bound, and it did.
Nobody can bound superintelligence once truly achieved. Not even the company that built its identity on knowing when to stop. The history of the atomic bomb’s development is milder than the movie’s interpretation.
Extract — continue reading at the source.
