Interviewing AI researchers on automation of AI R&D

## Introduction

The question of when and how AI might automate AI R&D is crucial for AI forecasting—if AI could automate the tasks involved in AI research, it could [drastically accelerate AI progress](https://epoch.ai/blog/do-the-returns-to-software-rnd-point-towards-a-singularity). There is a long history of researchers considering this question in the abstract, and describing its importance for how AI will shape the future.[1](https://epoch.ai/blog/interviewing-ai-researchers-on-automation-of-ai-rnd#user-content-fn-1) However, AI researchers disagree substantially on timelines for automating AI R&D—for instance, researchers’ predictions for [when all AI researcher tasks will be automated](https://arxiv.org/abs/2401.02843) vary between years and centuries.[2](https://epoch.ai/blog/interviewing-ai-researchers-on-automation-of-ai-rnd#user-content-fn-2)

In this work, we interviewed AI researchers with three goals:

1. Characterize AI R&D work tasks in detail, to better understand how automation might take place.
2. Clarify the reasoning underpinning researchers’ predictions about automation, to see where and why they disagree.
3. Collect their views on evaluations intended to measure how capable AI systems are at performing AI R&D, to better understand how society can track AI progress in this critical area.

To do this, we used qualitative interviews. We asked open-ended questions to eight AI researchers across industry, nonprofit and academic labs who have either published at leading conferences, or had similar experience. We identified recurring themes in their answers, and then summarized these, providing example quotations from participants where relevant.

Participants’ predictions for automation differed greatly, similar to pre-existing survey findings. However, all participants agreed engineering tasks will be the main driver of R&D automation in the next five years. If participants are correct, the question is _when_ engineering tasks will be automated, rather than _which_ tasks are relevant. If AI could solve existing AI R&D evaluations, focused on engineering tasks, most participants believed this would significantly accelerate their work.

## Summary of findings

- Creating hypotheses and planning research are vital for AI R&D, but researchers’ descriptions suggest they occupy relatively little time within a project. Meanwhile, engineering tasks such as coding and debugging are similarly important, but more time-consuming. Engineering tasks are central to participants’ work, even as they become more senior and take on more planning and management responsibilities.

- Predictions differ greatly for automation pace, but share a focus on engineering tasks as the driver of R&D automation in the near term. Differences in predictions for automation arise mostly from differences in timelines for when engineering tasks will be automated, rather than differing beliefs about which tasks are relevant.

- Existing R&D AI evaluations, focused on 

... (truncated, 19 KB total)