In 2025, human uplift studies face rising costs when conducted without AI tools. This post explores coding agent transcripts as an economical alternative for estimating productivity uplift using AI, specifically Claude Code transcripts generated in January 2026 by METR staff. The analysis, employing an LLM judge, reveals a time savings factor of approximately 1.5x to 13x for tasks executed with Claude Code, suggesting significant productivity boosts. However, this range may overstate true productivity due to factors like task substitution and selection effects, as AI is typically used for tasks perceived as lower-value. The concurrency of agent use also positively correlates with time savings, though not capturing a complete productivity picture. Limitations include the validation of LLM estimates and the narrow focus on a single month and specific team members. Ultimately, while AI-assisted tasks show substantial time savings, the actual productivity uplift may remain modest and warrants further research for refinement.
Source link
Share
Read more