For almost a decade, it's been my job to think about where AI is heading and what could go wrong, first at OpenAI, then DeepMind, then as Chief Scientist at the UK AI Security Institute (AISI). This experience has made it clear to me that we can't wait for all of our uncertainties to be settled to do something about AI's risks. Recent warnings about the potential destructive power of AI are understating the severity of the situation.
I believe there’s about a 50% chance we all die because of the development of smarter-than-human AI systems, and that our actions over the next two to 10 years will determine the outcome. To be clear, when I say our odds of being killed by superintelligent AI are 50%, I’m not claiming precision. Instead, I’m highlighting that several of the most fundamental debates about the future and safety of AI remain unresolved.
I believe we can’t expect these disagreements to be resolved until it’s too late to change course. We will have to act despite the uncertainty. And resist the temptation to divert urgent discussions toward thorny disputes.
Ultimately, AI’s capacities and motivations, and our own human actions, all feed into our likelihood of survival. Here’s why I estimate our odds are comparable to a coin flip—and why I believe the time to pause AI development is now. Let’s focus on everyone being killed by superintelligent AI for a moment.
Which capabilities would that require? My sense is that an AI system could kill us all with just four types of skills. To kill all humans, a superintelligent AI would need to take over and migrate between computer systems.
We saw this during the Hugging Face incident. Second, a superintelligent AI would need the ability to persuade, to manipulate humans into acting on its behalf, as we studied at AISI. Third, a lethal superintelligent AI would need to conceal its thoughts from us, which is increasingly common in the latest, most capable models.
Extract — continue reading at the source.
