Computer-use agents are AI systems that autonomously interact with software through graphical user interfaces (GUIs), handling tasks like form filling and workflow management. Despite their potential, these agents often struggle in real-world applications due to the “knowledge–action gap,” where even accurate external information doesn’t translate into effective actions. A recent study highlights that these agents achieve only a 41% success rate, even with 90% accurate instructions. To bridge this gap, Microsoft Research Asia developed UI-Evol, a component that enhances agent performance by leveraging the actual user interface for real-time updates and guidance. UI-Evol employs a two-stage process: retrace, which captures successful actions, and critique, which refines external instructions based on actual agent behavior. This innovation significantly improves agent reliability and performance, addressing variability in outcomes. As computer-use agents transition to practical applications like virtual assistants and robotic process automation, consistent reliability, as provided by UI-Evol, is essential for user trust.
