研究团队做了一个GUI探索器，自动找出那些需要你亲自操作的敏感界面，比直接让AI乱点安全多了。
该论文提出一个GUI探索器agent，从示范任务出发系统探索查询空间，识别会导致用户敏感状态的GUI操作。现有LLM agent通常被微调为不管安全影响都完成任务，难以部署。论文定义了用户敏感状态和查询的分类，帮助工程人员在关键场景下识别并请求用户接管。实验在开放GUI环境中验证了方法的有效性。
GUI agent: Guided Exploration of User-Sensitive Screens
LLM agents are increasingly being used to automate tasks for users within an open GUI environment. They inevitably encounter screens containing user-sensitive information, for which takeover of task execution by the user is highly desirable or even necessary. State-of-the-art LLM-driven agents are usually fine-tuned to complete tasks regardless of the safety implications of their actions. This makes their real-world deployment difficult and adversely affects the reliability. Therefore, it is crucial to identify and categorize user-sensitive states and define user-sensitive queries. This dataset would be to engineers to recognize and request handover to the user in critical scenarios. This short paper develops an explorer agent that systematically explores the query space starting from one demonstrated task to identify queries that, if executed, would lead to user-sensitive states in a GUI environment.