这篇论文解决了监控视频中事故理解的零样本难题,做视频分析或安全监控的开发者可以直接借鉴其三阶段分解思路,比传统提示方法更可靠,值得一试。
本文提出一种三阶段流水线,用于从监控视频中零样本理解事故,包括何时发生冲击、何种类型以及发生在画面何处。第一阶段通过视觉-语言相似性提取冲击附近的时间窗口;第二阶段利用元数据驱动多提示推理,结合五个互补视角(基线、运动、几何、对比和决胜)并通过熵门控成对裁决器解决分歧;第三阶段基于预测的事故类型和场景布局,使用开放词汇检测器定位冲击,并通过得分加权质心聚合关键帧检测结果。该方法在零样本ACCIDENT @ CVPR基准上显著优于中心帧基线,表明将零样本视频理解分解为时间定位、语义分类和空间定位能比直接提示更可靠地利用视觉-语言模型。
Metadata-Aware Multi-Prompt Reasoning for Zero-Shot Accident Understanding
In this paper, we address the problem of zero-shot understanding of accidents from surveillance videos by identifying when an impact event occurs, what type of impact it is, and where in the frame it occurs using natural language. We propose a three-stage pipeline that decomposes the accident understanding into when, what, and where. The first stage extracts a short temporal window around the impact using vision-language similarity. In the second stage, we perform metadata-driven multi-prompt reasoning with five complementary views (baseline, motion, geometry, contrast, and tiebreaker) and resolve disagreement via an entropy-gated pairwise adjudicator. Finally, we localize the impact of an open-vocabulary detector queried on the predicted accident type and scene layout, and aggregate detections across keyframes using a score-weighted centroid. Our pipeline achieves a substantial improvement in the harmonic-mean score over a centre-of-frame baseline on the zero-shot ACCIDENT @ CVPR benchmark. We show that decomposing zero-shot video understanding into temporal localization, semantic classification, and spatial grounding enable more reliable reasoning with vision-language models than direct prompting alone.