这项研究直击视频 AI 助手的核心痛点——实时纠错能力，做智能烹饪指导或边缘 AI 应用的开发者值得关注，Ego-CoMist 数据集可以直接用于微调模型。
研究人员提出了 Ego-MC-Bench 基准测试，用于评估视频大语言模型在实时任务指导中主动干预纠错的能力。该基准聚焦于烹饪场景，要求模型在用户犯错时及时介入。实验表明，当前最先进的视频 LLM 在此任务上表现不佳，主要原因是缺乏包含错误和适时干预的训练数据。为此，团队还创建了 Ego-CoMist 合成数据集，通过将非交互式烹饪视频转化为带干预的监督示例。微调该数据集后，小型高效视频 LLM 的性能显著提升，适合部署在边缘设备上提供实时辅助。
Streaming Interventions: Can Video Large Language Models Correct Mistakes as They Occur?
Learning everyday skills, like cooking a dish, relies increasingly on instructional media such as online videos. This opens the door to the use of video (and multimodal) large language models (LLMs) as task guidance assistants. A crucial capability for the real-world success of a prospective task guidance assistant is it's ability to intervene proactively as soon as a mistake is apparent in order to guide the user. To evaluate this crucial capability, we introduce Ego-MC-Bench (Mistake Corrections), a benchmark for evaluating reactive, step-by-step task guidance in realistic cooking scenarios. Extensive experiments show that Ego-MC-Bench is highly challenging for state-of-the-art video LLMs. We argue that a key reason is the limited availability of training data for fine-tuning models on this task. Although there exists a wide range of cooking video datasets, existing datasets lack examples of mistakes along with appropriately timed interventions. To help address this data limitation, we also introduce Ego-CoMist, a counterfactual synthetic dataset created by transforming non -interactive cooking videos into supervised training examples showing proactive interventions. We show that fine-tuning on Ego-CoMist yields performance gains especially for smaller and more efficient video LLMs that are well suited for delivering assistance on edge devices.