ComFair 基准上线：测试 LLM 多重属性交叉公平性
Beyond Single-Attribute Auditing: A Configurable Multi-Attribute Benchmark for Intersectional Fairness in LLMs
一个测交叉公平性的新基准，10 万级可控实例能把单属性测试里看不出的问题暴露出来，做模型审计的可以试试。
ComFair 是一个可配置的公平性审计基准，把已有公平性数据转换成 479,066 个受控的双属性和多属性测试实例，并保留任务等价的原始单属性版本。它覆盖 Recognition、Selection、Continuation、Conversation 四类任务，可统一测量性能变化、偏差分数变化，以及新增属性引入或掩盖错误的比例。在十种 LLM 上的实验显示：判别任务能暴露两个方向的转变，生成任务的偏移更大且方向不对称，模型排名会随属性组合而改变。该基准用于检验公平性结论在多属性交叉场景下是否依然成立。
Beyond Single-Attribute Auditing: A Configurable Multi-Attribute Benchmark for Intersectional Fairness in LLMs
Large language models (LLMs) are increasingly evaluated for fairness, yet most existing benchmarks audit one sensitive attribute at a time. This single-attribute paradigm can miss intersectional harms, where multiple attributes interact non-additively and produce failures that are invisible under marginal evaluation. We introduce ComFair, a configurable benchmark that converts existing fairness data into 479,066 controlled Dual- and Multi-attribute instances and pairs every augmented instance with its task-equivalent source. Across Recognition, Selection, Continuation, and Conversation, this pairing supports a unified audit of performance or bias-score shifts, how often added attributes introduce new errors or mask existing ones, and model-ranking stability. Experiments on ten LLMs show task- and subgroup-dependent sensitivity: discriminative tasks expose both transition directions even when aggregate changes are modest, generative tasks exhibit larger and directionally asymmetric shifts, and rankings vary with attribute composition. ComFair therefore provides a scalable controlled sensitivity audit---rather than a model of the natural joint distribution of identities---for testing the robustness of fairness-benchmark conclusions beyond single-attribute settings.
