Papers

Social Signal Bench:47小时音视频社交信号评测基准发布

Social Signal Bench: Evidence-Weighted Evaluation of Co-occurring Social Signals in Video and Audio

Why we picked it

做对话分析、客服质检或者情感计算的同学可以看看这个基准。它把兴趣、犹豫这类容易混淆的共现信号单独拎出来评,还配了个比普通 F1 更讲道理的 EW-F1 指标。7B 开源模型微调后能打过前沿闭源模型,说明这个方向自己练小模型完全可行。

研究团队发布 Social Signal Bench,一个总计 47 小时、稠密标注的社交信号识别评测基准,包含 25 小时的视频数据集(覆盖 10 种社交信号)和 22 小时的音频数据集(覆盖 6 种信号)。

Body

训练数据来自在线对话视频,其中 10 小时经专家标注,其余为伪标签,测试集由专家标注并覆盖四个对话领域,且不公开。论文同时提出 Evidence-Weighted F1(EW-F1)指标,按标注者证据为参考标签加权,并通过逐片段最优匹配给语义相关的预测赋予梯度得分,扰动实验显示它能区分语义相关错误与无关错误。

在统一评测协议下,基于该基准后训练的 7B 开源模型在视频任务上超过通用前沿模型,在音频任务上与之持平。

Original · arXiv cs.LG

Social Signal Bench: Evidence-Weighted Evaluation of Co-occurring Social Signals in Video and Audio

Recognizing social signals such as interest and hesitation requires interpreting context-dependent behaviors that may co-occur in conversation. Existing affective computing benchmarks rarely evaluate such co-occurring signals, and conventional multi-label F1 overlooks semantic relationships and annotator disagreement. We introduce Social Signal Bench, a 47-hour densely annotated benchmark for the evaluation of social signal recognition and engagement, comprising two datasets: Social Signal Bench Video (25 hours), covering ten social signals in audiovisual recordings, and Social Signal Bench Audio (22 hours), covering six signals in audio-only recordings. Each dataset pairs a public training set of online conversational video, of which 10 hours are expert-annotated and the remainder pseudo-labeled, with a private expert-labeled test set spanning four conversational domains. We also propose Evidence-Weighted F1 (EW-F1), which weights reference labels by annotator evidence and assigns graded credit to semantically related predictions through optimal per-clip matching. Controlled perturbations demonstrate that EW-F1 distinguishes semantically related errors from unrelated ones, unlike conventional F1. Under a shared evaluation protocol, 7B open models post-trained on Social Signal Bench outperform general-purpose frontier models on video and match them on audio.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include Social Signal Bench, EW-F1, affective computing, multimodal.

Source attribution: curated from arXiv cs.LG (@Siddharth Ravi, Anders R. Bargum, Hugo Anselme, Patrizio Cugia di Sant'Orsola, Alberto Cardaci, Flora Kincso Parragi, Vincent Borowiec, Line H. Clemmensen).