“Assessing AI Agent Success Rates: Is There a ‘Half-Life’?”

The study explores how AI agents’ performance declines with task duration, estimating task lengths corresponding to a 50% success rate. Interestingly, this task length has been doubling every seven months, reflecting advancing capabilities. Researchers also measured the 80% success rate, which had a similar doubling time of about 213 days. However, significant differences emerged: task lengths for 80% success rates were substantially shorter. For instance, the best model achieved a 50% success rate on tasks lasting 59 minutes, compared to only 15 minutes for an 80% success rate. This leads to the conclusion that the task length for achieving an 80% success rate is roughly one-fourth that of the 50% rate. The study’s innovative approach offers a clearer improvement measure across diverse tasks, though it raises questions about its generalizability beyond the tested suite. The findings suggest that longer tasks may share a common hazard rate, akin to constants observed in survival analysis.

Source link