Researchers highlight significant flaws in existing skill-testing benchmarks, such as SKILLSBENCH, which provide agents with curated, task-specific skills, essentially offering direct solutions. In contrast, real-world scenarios require agents to navigate noisy, unstructured skill collections independently. The study assessed 34,198 real skills from open-source platforms like skillhub.club and skills.sh, revealing performance drops across six increasingly realistic testing scenarios. For instance, Claude Opus 4.6’s pass rate decreased from 55.4% with curated skills to just 38.4% without them. Lesser models like Kimi K2.5 fared worse, even falling below their no-skill baseline. Key challenges identified include poor selection of skills, ineffective search strategies, and failures in adapting general skills. Although task-specific refinement showed promise, task-independent refinement yielded minimal improvements. The study urges better retrieval methods and skill ecosystems tailored for diverse models, emphasizing the need for enhanced performance to facilitate effective skill utilization in AI agents.
Source link
Share
Read more