arXiv 论文实证审计:LLM 简历筛选系统的部署就绪度
Can AI Judge You? An Empirical Audit of Deployment Readiness in LLM-assisted Hiring Systems
四位研究者拿210份简历实测了4个LLM做简历筛选的靠谱程度,发现打分比通过/不通过更准,模型的决定和理由还经常对不上。想做AI招聘的企业和HR工具开发者,这篇给出了能直接抄的审计清单。
一篇 arXiv 论文构建了将候选人证据、评估、权限控制和评估记录相互分离的简历筛选系统架构,并对两个维度的部署就绪度做实证检验。实验在 4 个 LLM 上进行,使用 210 份简历、14 个岗位角色和 6 种决策接口。结果显示,标量分数比简单的通过/不通过判断更能区分资质变化,而比较型判断对呈现顺序敏感。
合规维度上,显式权限规则和将简历内容与指令分离能降低未授权请求的通过率,但在不同 LLM 和使用场景下效果不一。论文还发现 LLM 评估器给出的决定与其提供的理由经常不一致。
论文将评估对象从单个模型扩展到完整配置的简历评估系统,为企业在部署前提供审计框架。对 HR 科技厂商和采用 LLM 做招聘筛选的企业来说,接口设计和权限隔离的细节直接影响实际筛选质量。
Can AI Judge You? An Empirical Audit of Deployment Readiness in LLM-assisted Hiring Systems
Delegating resume screening to large language models (LLMs) requires organizations to establish whether they can distinguish candidates by relevant qualifications while allowing only authorized users to change screening rules or approve exceptions. To this end, we develop a system architecture that separates candidate evidence, evaluation, permission controls, and assessment record, and empirically examine two complementary dimensions of its deployment readiness. First, we assess its operational readiness on four LLMs using a controlled experiment on 210 resumes across 14 job roles and six decision interfaces. Second, we assess its compliance readiness by testing how it handles unauthorized user requests and instructions embedded in resumes. We find that operational performance depends on the choice of evaluation interface: scalar scores generally distinguish qualification changes better than simple pass/fail judgments, while comparative judgments are sensitive to presentation order. Explicit permission rules and separation of resume content from instructions reduce unauthorized compliance, to a varying degree of effectiveness across LLMs and usage contexts. Finally, we find that LLM evaluators often diverge between their decisions and offered justifications. Going beyond simply evaluating LLMs for hiring tasks, our framework supports an audit of a fully configured resume evaluation system to enable its deployment decision.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include LLM, AI hiring, resume screening, audits.
Source attribution: curated from arXiv cs.AI (@Spandan Roy, Subhabrata Majumdar, Soudeep Deb).