Ajeya Cotra on whether it's crazy that every AI company's safety plan depends on AGI being decades away

On this page:

 1 Introduction 1.1 The episode in a nutshell 
 
 2 Highlights 
 3 Articles, books, and other media discussed in the show 
 4 Transcript 4.1 Cold open [00:00:00] 
 4.2 Ajeya's strong track record for identifying key AI issues [00:00:41] 
 4.3 The 1,000-fold disagreement about AI's effect on economic growth [00:02:31] 
 4.4 Could any evidence actually change people's minds? [00:23:26] 
 4.5 The most dangerous AI progress might remain secret [00:30:39] 
 4.6 White-knuckling the 12-month window after automated AI R&D [00:47:21] 
 4.7 AI help is most valuable right before things go crazy [01:12:01] 
 4.8 Foundations should go from paying researchers to paying for inference [01:24:42] 
 4.9 Will frontier AI even be for sale during the explosion? [01:32:03] 
 4.10 Pre-crunch prep: what we should do right now [01:43:59] 
 4.11 A grantmaking trial by fire at Coefficient Giving [01:47:03] 
 4.12 Sabbatical and reflections on effective altruism [02:07:45] 
 4.13 The mundane factors that drive career satisfaction [02:37:17] 
 4.14 EA as an incubator for avant-garde causes others won't touch [02:46:55] 
 
 5 Learn more 
 6 Related episodes 
 Read transcript See all episodes 
 
 
 
 Every major AI company has the same safety plan: when AI gets crazy powerful and really dangerous, they’ll use the AI itself to figure out how to make AI safe and beneficial. It sounds circular, almost satirical. But is it actually a bad plan?

 Today’s guest, Ajeya Cotra, recently placed 3rd out of 413 participants forecasting AI developments and is among the most thoughtful and respected commentators on where the technology is going.

 She thinks there’s a meaningful chance we’ll see as much change in the next 23 years as humanity faced in the last 10,000, thanks to the arrival of artificial general intelligence. Ajeya doesn’t reach this conclusion lightly: she’s had a ring-side seat to the growth of all the major AI companies for 10 years — first as a researcher and grantmaker for technical AI safety at Coefficient Giving ( formerly known as Open Philanthropy ), and now as a member of technical staff at METR .

 So host Rob Wiblin asked her: is this plan to use AI to save us from AI a reasonable one?

 Ajeya agrees that humanity has repeatedly used technologies that create new problems to help solve those problems. After all:

 Cars enabled carjackings and drive-by shootings, but also faster police pursuits.
 Microbiology enabled bioweapons, but also faster vaccine development.
 The internet allowed lies to disseminate faster, but had exactly the same impact for fact checks.
 But she also thinks this will be a much harder case. In her view, the window between AI automating AI research and the arrival of uncontrollably powerful superintelligence could be quite brief — perhaps a year or less. In that narrow window, we’d need to redirect enormous amounts of AI labour away from making AI smarter and towards alignment research, biodefence, cyberdefence, adapting our p

... (truncated, 98 KB total)