Papers

ADDIS 方法修正 LLM 交叉性偏见审计中的假阳性问题

LLM GerryMandering: Separating Interactions from Additive Effects in LLM Audits

Why we picked it

做模型公平性审计的人值得看这篇:现有审计标准在合成数据上平均每数据集误报 66 个子群体,基本是在测单属性偏差。ADDIS 用加性基线把误报压到 0.09,还能回收 84% 的真实交互。审计团队可以直接借鉴这套方法。

arXiv 论文 2611.26538 提出 ADDIS(Additive Intersectional Screen)方法,用于修正大模型交叉性偏见审计中的假阳性。作者在 150 个无交互效应的合成数据集上测试现有审计常用标准:采用 shuffle 零假设时平均每个数据集误报 66.1 个子群体,且每个数据集至少误报一次,说明独立属性效应本身就足以触发该标准。

Body

ADDIS 的机制是对所有属性拟合加性模型,用每个子群体的 Additive Residual Distance(ARD)衡量其偏离加性预测的程度,再用 Freedman-Lane 零假设做检验。在同一合成基准上,ADDIS 平均每个数据集仅误报 0.09 个子群体,并回收了 84% 被植入的已知交互效应,而修复后的零假设标准只能回收 32%。

作者构建 IndiPersona 数据集(10,000 个含六项身份属性的印度合成人设)审计 13 个开源语言模型。旧标准平均标记 738 个测试子群体中的 7.1%,ADDIS 仅标记 3.0%。性别认同与性取向的交叉在所有模型中都是最大的交互配对,反事实人设测试还在除一个模型外的全部模型中发现了观察性测试无法检测的种姓与阶级交互。

Original · arXiv cs.AI

LLM GerryMandering: Separating Interactions from Additive Effects in LLM Audits

Intersectional bias arises when a model treats people who hold several marginalised identities at once differently from what their separate identities would predict. Auditing such intersectional subgroups remains a challenge: their number grows quadratically, and there is no agreed baseline for what independent attributes would predict. Without such a baseline, many existing audits flag a subgroup when it exceeds each attribute alone, a condition that attributes acting independently already satisfy. We test this standard criterion on 150 synthetic datasets built with no interaction. With the shuffle null, it flags 66.1 false subgroups per dataset on average, and at least one in every dataset. When we plant interactions of known size into these datasets, a repaired null lets the criterion recover only 32% of them. We introduce ADDIS (Additive Intersectional Screen), which fits an additive model over all attributes, scores each subgroup by its Additive Residual Distance (ARD) from that model's prediction, and tests it against a Freedman-Lane null. On our synthetic benchmark, ADDIS flags 0.09 false subgroups per dataset and recovers 84% of the same planted interactions. We then build IndiPersona, 10,000 synthetic Indian personas with six identity attributes, and audit 13 open language models. The standard criterion flags 7.1% of the 738 tested subgroups on average; ADDIS flags 3.0%. Gender identity crossed with sexual orientation is the largest pairing in every model, and counterfactual personas confirm it and additionally reveal a caste x class interaction in all but one model that the observational test lacked the data to detect. Much of what audits report as intersectional bias can thus arise from independent single-attribute effects alone, yet a small core of interactions survives.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include LLM, fairness, ADDIS, IndiPersona.

Source attribution: curated from arXiv cs.AI (@George Rahul, Ronit Jahilal, Bhaskara Hanuma Vedula, Ponnurangam Kumaraguru, Pushkar Shukla).