Two of the “godfathers” of modern AI and senior executives at OpenAI and Anthropic have warned governments to prepare for an AI “intelligence explosion”, which they say could be the most consequential technological development in history. A report co-authored by the Nobel laureate Geoffrey Hinton and the Canadian computer scientist Yoshua Bengio, considered godfathers of modern AI for their work in the field, urges politicians to act now before there is runaway progress in the technology. The authors say they are concerned about an “intelligence explosion”, which they describe as a “dramatic AI-driven acceleration of AI progress, compressing advances that would otherwise take years into months or less”.
The paper, titled “What if automating AI R&D triggers an intelligence explosion”, is authored by more than 20 people including Hinton and Bengio. Other authors of the paper include Jack Clark, a co-founder of Anthropic, and Jakub Pachocki, the chief scientist at OpenAI. Their concerns focus on the possibility of AIs being able to improve themselves without human intervention, a process known as “recursive self-improvement”.
In the paper it is broadly referred to as automated AI research and development. Once AI systems reach expert-level capabilities at AI R&D, a single developer could run a workforce equivalent to “millions” of top human researchers, the paper says. The authors say the automating of AI R&D is the most likely source of an intelligence explosion, because AIs are already contributing to improving their own technology and the resulting improved systems can be rapidly deployed once built.
Urging governments to take action quickly, the authors say: “Once an intelligence explosion begins, the window for action may close.” They recommended a trio of policy priorities: requiring transparent progress reports on AI-related R&D, including embedding independent auditors in companies; finding ways to constrain breakneck AI development; and preparing to adapt to an intelligence explosion. Anthropic and OpenAI have both agreed to having independent evaluators assess their models after warning that AI development was reaching a critical pitch in terms of safety. Anthropic says AI now produces 80% of its own code, while OpenAI uses autonomous AI agents in areas such as the training of new models.
While this could produce medical breakthroughs and technological leaps, it would bring a trio of risks: powerful systems could enable biological and cyber threats that develop faster than the measures needed to counter them; as humans become less involved in AI R&D they could lose the opportunity to control systems; states could use an intelligence explosion to convert a modest lead in areas such as cyberspace into a decisive one. The report acknowledges these impacts are “uncertain”. For instance, a technological breakthrough could take time to implement given the need to set up supply chains for any special materials involved or to comply with regulations.
AI systems could also accelerate the speed at which risks are mitigated, say the authors. If such a breakthrough happened, the authors say, AI systems could rapidly eclipse human experts in most fields and “radically accelerate” technological progress. Preparing for this “should be an urgent priority, including at the highest levels of government leadership”.
Extract — continue reading at the source.
