Social Signal Bench:47小时音视频社交信号评测基准发布
Social Signal Bench: Evidence-Weighted Evaluation of Co-occurring Social Signals in Video and Audio
做对话分析、客服质检或者情感计算的同学可以看看这个基准。它把兴趣、犹豫这类容易混淆的共现信号单独拎出来评,还配了个比普通 F1 更讲道理的 EW-F1 指标。7B 开源模型微调后能打过前沿闭源模型,说明这个方向自己练小模型完全可行。
研究团队发布 Social Signal Bench,一个总计 47 小时、稠密标注的社交信号识别评测基准,包含 25 小时的视频数据集(覆盖 10 种社交信号)和 22 小时的音频数据集(覆盖 6 种信号)。
训练数据来自在线对话视频,其中 10 小时经专家标注,其余为伪标签,测试集由专家标注并覆盖四个对话领域,且不公开。论文同时提出 Evidence-Weighted F1(EW-F1)指标,按标注者证据为参考标签加权,并通过逐片段最优匹配给语义相关的预测赋予梯度得分,扰动实验显示它能区分语义相关错误与无关错误。
在统一评测协议下,基于该基准后训练的 7B 开源模型在视频任务上超过通用前沿模型,在音频任务上与之持平。
Social Signal Bench: Evidence-Weighted Evaluation of Co-occurring Social Signals in Video and Audio
Recognizing social signals such as interest and hesitation requires interpreting context-dependent behaviors that may co-occur in conversation. Existing affective computing benchmarks rarely evaluate such co-occurring signals, and conventional multi-label F1 overlooks semantic relationships and annotator disagreement. We introduce Social Signal Bench, a 47-hour densely annotated benchmark for the evaluation of social signal recognition and engagement, comprising two datasets: Social Signal Bench Video (25 hours), covering ten social signals in audiovisual recordings, and Social Signal Bench Audio (22 hours), covering six signals in audio-only recordings. Each dataset pairs a public training set of online conversational video, of which 10 hours are expert-annotated and the remainder pseudo-labeled, with a private expert-labeled test set spanning four conversational domains. We also propose Evidence-Weighted F1 (EW-F1), which weights reference labels by annotator evidence and assigns graded credit to semantically related predictions through optimal per-clip matching. Controlled perturbations demonstrate that EW-F1 distinguishes semantically related errors from unrelated ones, unlike conventional F1. Under a shared evaluation protocol, 7B open models post-trained on Social Signal Bench outperform general-purpose frontier models on video and match them on audio.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include Social Signal Bench, EW-F1, affective computing, multimodal.
Source attribution: curated from arXiv cs.LG (@Siddharth Ravi, Anders R. Bargum, Hugo Anselme, Patrizio Cugia di Sant'Orsola, Alberto Cardaci, Flora Kincso Parragi, Vincent Borowiec, Line H. Clemmensen).