arXiv 论文提出单体化鲁棒学习:单数据集训练多个假设降低联合失效概率
Monoculture Robust Learning
搞理论或关心模型同质化风险的人可以看这篇。它把 2022 年的可复制性理论和 2021 年的算法单体化问题接上了:同一个数据集跑 k 次可复制学习器,联合出错率就能指数级下降,代价是样本量要多 Ω(log(1/γ)) 倍。比独立采数据集省钱,思路对做集成和风险分析的人有直接参考价值。
一篇 arXiv 论文(编号 2611.24881)在 PAC 学习框架下提出"单体化鲁棒性"概念,针对 Kleinberg 和 Raghavan(2021)指出的算法单体化问题:共享训练数据的模型往往在同一批输入上同时出错。论文的核心问题是:仅使用一个数据集的学习器能否输出 k 个各自准确、且联合失效概率接近用 k 个独立数据集训练的假设。
技术上,作者建立了可复制性(Impagliazzo 等,2022)与单体化鲁棒性之间的联系:若基础学习器满足 ρ-可复制,则在同一样本上独立运行 k 次,个体单体化误差会随 k 指数级下降,下降速度由基础学习器的单次误差和 ρ 控制。
论文同时给出下界:要把个体联合误差压到 γ,样本量需要乘以 t = Ω(log(1/γ)) 的因子,说明该方法存在固有的样本代价。
Monoculture Robust Learning
We introduce a notion of monoculture robustness in Probably Approximately Correct (PAC) learning, as a way to mitigate the problem of algorithmic monoculture (Kleinberg and Raghavan, 2021): the tendency of models sharing the same components (e.g., training data) to fail on the same inputs. Specifically, we ask whether a learner with access to a single dataset can output $k$ individually accurate hypotheses whose joint failure probability is comparable to training on $k$ independent datasets. We say that such a learning procedure achieves monoculture robustness, and we quantify this by bounding the conjunctive error that all $k$ hypotheses misclassify points in an individual sense. We design methods for achieving this type of guarantee by drawing a surprising connection between replicability (Impagliazzo et al., 2022) and monoculture robustness. Specifically, we show that if the base learner is $ρ$-replicable, then running it independently $k$ times on the same sample yields individual monoculture error that shrinks exponentially with $k$, controlled by the single-run error of the base learner at the point and $ρ$. We complement these positive results with a lower bound: for individual monoculture robustness, we prove a worst-case lower bound showing that, for learners, driving the individual conjunctive error down to $γ$ can require a sample-size multiplier of $t = Ω(\log(1/γ))$.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Key entities and technical topics under continuous tracking include PAC learning, machine learning theory.
Source attribution: curated from arXiv cs.LG (@Lee Cohen, Jon Kleinberg, Omer Reingold).