Jacob Coxon published the viral post from a park bench in San Francisco’s Alamo Square. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.” Coxon, a 27-year-old British researcher, hit send on X with no expectations.
Instead, he became the unlikely catalyst of a chain reaction that changed the debate over artificial intelligence. Within 36 hours, Coxon’s original posts received 153 million views. Dozens of politicians joined the chorus of concern.
On Sept. 12, Anthropic CEO Dario Amodei published a 3,800-word essay arguing that the mounting risks of AI warranted a slowdown. In as little as six to 12 months, he wrote, misaligned AIs “could be capable of taking over the entire internet.” Two of his fiercest rivals, Elon Musk of SpaceXAI and Sam Altman of OpenAI, said they agreed decelerating was necessary; Altman announced that because of safety concerns his company wouldn’t go public this year. Both he and Amodei added they were willing to invite independent safety experts inside their companies to help stem the risks that AI escapes human control, though stopped short of announcing they would immediately slow down or stop the training of new models.
Read More: He Helped Build Powerful AI at OpenAI and Anthropic. Now He's Afraid It Could Kill Us Coxon’s warning was far from the first time an AI safety expert issued an apocalyptic missive. But it landed at the end of a summer that showcased how frighteningly capable AI has become.
Frontier models from Anthropic and OpenAI can now solve the world’s hardest math problems and crack cutting-edge cybersecurity systems. Those models are increasingly used to accelerate the development of more powerful successors. No less an authority than Nvidia CEO Jensen Huang declared that artificial general intelligence—the threshold at which AI matches or surpasses most human capabilities—has arrived.
Some fear we are now on the cusp of a tipping point, where AI systems could improve themselves autonomously, a phenomenon known as recursive self-improvement. At the same time, a series of unprecedented safety incidents offered unsettling evidence that the companies building powerful AI systems could not always control them. In July, OpenAI announced that a swarm of its agents had hacked a separate company, Hugging Face, after going rogue during a cybersecurity test.
Extract — continue reading at the source.
