In the summer of 1974, scientists who had found the ability to rearrange the essence of life hit the emergency stop. Recombinant DNA (combining genetic material from different organisms) was so new that they worried about world-changing consequences if an engineered organism escaped from a lab. A committee led by Stanford biochemist Paul Berg recommended that experiments be halted until scientists worked out how to conduct them safely.
The following February, around 140 scientists gathered at the Asilomar Conference Center in California and eventually thrashed out safeguards so that research could continue. The National Institutes of Health (NIH) turned those into formal guidelines in 1976 and made federal funding conditional on following them. Half a century later, a researcher at one of the world’s leading tech companies has reached a version of the same conclusion, but too late.
Jacob Coxon, 27, worked at OpenAI and then at Anthropic. On September 8 he resigned. They are, he said, racing straight to self-improving superintelligence and “gambling with our lives.” He asked the labs to coordinate and called for a temporary ban on making models more capable.
It was the Berg letter redux. AI had its own Asilomar moment nearly a decade ago, and squandered it. In 2017, just like in 1975, researchers gathered at Asilomar for the Beneficial AI conference, convened by the Future of Life Institute, and produced 23 principles intended to guide the development of advanced artificial intelligence.
Principle No. 5 was called “Race Avoidance.” It was simple: “Teams developing AI systems should actively cooperate to avoid corner-cutting on safety standards.” AI did not fail to have an Asilomar moment. It had one, and everyone went racing anyway. Evan Hubinger, who leads alignment stress-testing at Anthropic, responded to Coxon by saying he was correct.
Hubinger wrote that he personally assigns a probability greater than 10 percent to AI killing all humans within the next decade, and that Anthropic does not yet have a plan to solve alignment for superintelligence and is “not clearly on track to.” Political concern is moving just as fast. On September 3, Democratic Representative Greg Casar and independent Senator Bernie Sanders announced the Ban Artificial Superintelligence Act, which would prohibit superintelligent systems and pause advanced AI development until a federal regulator has written safety rules. The coalition backing a ban is bipartisan, and it includes Geoffrey Hinton, Yoshua Bengio, Steve Wozniak, Richard Branson, Steve Bannon and Glenn Beck.
Extract — continue reading at the source.
