Evaluating AI Agents in Research: Insights from the Deep Research Bench Report

As large language models (LLMs) advance, they are increasingly marketed as powerful research assistants capable of undertaking complex tasks involving multi-step reasoning and data synthesis. Major players like OpenAI, Anthropic, Google, and Perplexity are developing features branded as “Deep Research” and other variations. A report from FutureSearch, titled Deep Research Bench (DRB), evaluates these AI agents’ performance on 89 challenging, web-based research tasks. OpenAI’s model, o3, emerged as the top performer, scoring 0.51, highlighting that even high scorers fall short of human researchers. Common issues include forgetfulness, repetitive searches, and incomplete conclusions, particularly detrimental in complex tasks. Interestingly, “toolless” models performed comparably in simpler tasks, revealing strong internal capabilities, though they struggled with more demanding queries. Overall, while advanced LLMs can surpass average humans on specific tasks, they still lag behind skilled researchers, particularly in adapting and reasoning throughout complex research processes.

Source link