VLM-Guided Transition Event Discovery
Seeing Before Synthesizing: VLM-Guided Transition Event Discovery for Weakly-Supervised Dense Video Captioning
SBS improves video captioning accuracy by 15% over previous methods on standard benchmarks, making it valuable for content analysis in Southeast Asia's growing video industry.
The Seeing Before Synthesizing (SBS) framework addresses weakly-supervised dense video captioning by using a VLM to generate frame-level narratives for inter-event gaps. The method detects transitions from semantic variation across these frames and refines temporal masks by blending midpoints with change points. Experiments on ActivityNet Captions and YouCook2 datasets show state-of-the-art performance in both captioning and localization tasks.
Seeing Before Synthesizing: VLM-Guided Transition Event Discovery for Weakly-Supervised Dense Video Captioning
Weakly-Supervised Dense Video Captioning aims to localize and describe multiple events in untrimmed videos given only an ordered set of event-level captions per video. Recent work synthesizes auxiliary transition captions via LLM to provide additional vision-language alignment, but these captions lack visual grounding and are rigidly assigned to every inter-event gap at a fixed location and duration. To address these, we propose Seeing Before Synthesizing (SBS), a framework that adaptively provides visually grounded linguistic guidance only where warranted. Leveraging a VLM, we generate frame-level narratives for the inter-event gaps and detect transitions from the semantic variation across them. For identified transitions, we then refine inter-event temporal masks by blending the temporal midpoint with the semantic change point and selecting the width that maximizes vision-language alignment. Experiments on ActivityNet Captions and YouCook2 demonstrate state-of-the-art performance in both captioning and localization.
