Carnegie Mellon University — Benchmark for Web-Based Tasks

[Navigating Transformative AI] Open Philanthropy recommended a grant of $547,452 to Carnegie Mellon University to support research led by Professor Graham Neubig to develop a benchmark for the performance of large language models conducting web-based tasks in the work of software engineers, managers, and accountants. This grant was funded via a request for proposals for projects benchmarking LLM agents on consequential real-world tasks. This falls within our focus area of potential risks from advanced artificial intelligence.