RecSys-HR 2026 WorkRB 挑战赛：用分级相关性评测技能抽取模型
Grading the Unannotated: Insights from the WorkRB Skill Extraction Challenge at RecSys-HR 2026
技能抽取评测从打分制换成了分级制，结果显示旧的二元榜单排名和分级排名完全不相关，做招聘语义匹配的可以看看这套标注方法。
WorkRB Challenge 在 RecSys-HR 2026 中放弃二元标注，将 ESCO 词汇表中 13,939 项技能的每条候选赋予 0 到 4 的相关性等级，并以 nDCG@100 排名。由于三元判断过粗，标注流程被拆解为四个二分类问题（同领域、活动提及、粒度正确、可替代 gold positive），由较小的 LLM 评委级联回答，并用盲评人类与前沿模型复核。比赛共构建 5 个基准数据集，参赛队伍普遍采用多阶段 retrieve-and-rerank 管线，其收益主要来自清除排名窗口中的语义无关预测。榜首的分级排序结果与二元评测结论不相关，且所有可测的参赛系统参数量至少是基线的 42 倍，通常达数百倍。
Grading the Unannotated: Insights from the WorkRB Skill Extraction Challenge at RecSys-HR 2026
Skill extraction benchmarks score models against a few human-annotated positives and treat every other skill in the vocabulary as equally wrong. At the scale of ESCO (13,939 skills) this binary view can reward models that rank the annotated positive high for the wrong reason, and hides semantically nonsensical predictions. The WorkRB Challenge at RecSys-HR 2026 replaces it with graded relevance: every candidate skill in a model's ranking window carries a 0 to 4 grade, and the leaderboard scores nDCG@100 against these grades. Since ternary judgements proved too coarse for objective annotation, relevance is decomposed into a cascade of four binary questions (same domain, activity mentioned, right granularity, replacement of a gold positive) that a smaller LLM judge can answer reliably; the labels are validated with blind human and frontier-model re-annotation. This paper documents the annotation methodology, the construction of five benchmark datasets (four with vacancy sentences as queries, one with lexically distant skill surface forms), the validation of the LLM cascade, and insights from the submissions. The field converged on multi-stage retrieve-and-rerank pipelines, whose gains come from clearing nonsense out of the ranking window rather than finding more positives: at the top of the leaderboard, the graded ordering is uncorrelated with what binary evaluation would have said. Graded labels expose much bigger gaps between top submissions and baselines than binary evaluation does, and show that semantic-understanding skill recommendation is still wide open. The clear next direction is bringing efficiency back: every submitted system we could size uses at least 42x, typically several hundred times, more parameters than the baselines it beats.
