As a California state senator, Scott Wiener authored legislation aimed at curbing the risks of advanced AI. Now, after winning a tight congressional race for former Speaker of the House Nancy Pelosi’s open San Francisco seat, he could play a key role in setting federal AI policy. I spoke with Wiener on Friday, only a few hours after he had been declared the victor.
But he was already thinking about AI risk. The previous day, the AI nonprofit METR had released a report that had been widely interpreted as evidence that Anthropic had achieved recursive self-improvement, a phenomenon in which AI meaningfully contributes to its own advancement and AI progress potentially spins out of control. SB 1047, a bill that he authored in 2024, was set to become the first real piece of frontier AI legislation in the country before it was vetoed by California governor Gavin Newsom; he then sponsored SB 53, a lighter-touch bill that Newsom signed.
Both bills have inspired proposed federal legislation. Wiener’s congressional campaign received over $3 million from Public First Action, a nonprofit in favor of AI regulation that receives substantial funding from Anthropic. That contribution led opponents to criticize him for being too close to the AI industry, but Wiener says that couldn’t be further from the truth.
And he’s enthusiastic about many of the policy ideas floating around Congress. He also highlights the importance of whistleblower protections, testing and auditing requirements, and a kill switch requirement, and he wants to see Congress leverage its oversight powers to hold the AI labs to account. And as a longtime state legislator, he is opposed to broad federal preemption of state AI laws, which some of the proposed federal legislation would require.
That means taking seriously the possibility of AI catastrophe, something he discusses in the sort of language typically used by AI safety experts. Once you’re in that realm, that really is potentially a recipe for total loss of control.” He’ll have some allies in the fight against catastrophic AI risk. Senator Bernie Sanders, for instance, has met with many of the AI safety movement’s most influential figures and has spoken repeatedly about AI and existential risk.
But there’s still a large gap between Wiener’s Bay Area, AGI-pilled perspective and the national mood on AI, which is anchored by opposition to data centers. The fact that voters approved 40 out of 41 anti-data-center ballot measures on Tuesday might seem to be a good sign for Wiener and others who support AI regulation, but Wiener says he sees this movement as far more tied to local concerns than to bigger questions about AI risk. I think that’s the bigger issue for most people, from what I can tell.” Wiener describes himself as only “cautiously optimistic” about Congress’s ability to rein in AI, given its poor track record on regulating the tech industry.
Extract — continue reading at the source.
