RACER Framework Routes Queries Between AI Models and Human Experts
RACER: Role-Aligned Competence Estimation for Human-AI Routing
Compared with L2D-Pop and IFD, RACER captures instance-level expert specialization without class-identity shortcuts, a practical design for medical AI triage systems that must route chest X-rays between models and radiologists.
RACER (Role-Aligned Competence Estimation for Routing) is a deferral framework that decides when a predictive system should act autonomously or hand a query to an unseen human expert, improving on prior methods L2D-Pop and Identity-Free Deferral (IFD), which either learn class-identity shortcuts or keep competence estimates constant within each class.
RACER estimates the posterior-predictive probability that an expert is correct on a query under each candidate class role, combines these with the model posterior, and comes with a proof of class-relabelling invariance plus a plug-in regret bound.
On synthetic benchmarks, it gives the strongest aggregate performance on a separately sampled unseen-expert split of CIFAR-100 and benefits from additional context in a PathMNIST histopathology context-scaling study with simulated experts.
On the chest-radiography benchmarks VinDr-CXR and CheXpert with radiologists, the RACER family is competitive or best in budget-swept deferral, with calibration results varying across metrics and datasets.
RACER: Role-Aligned Competence Estimation for Human-AI Routing
Learning to defer asks a predictive system when to act autonomously and when to defer to a human expert. Population-adaptive deferral extends this problem to unseen experts using a small context set of expert behavior. Neural context encoders such as L2D-Pop can be query-dependent, but may learn routing shortcuts tied to absolute class coordinates. Identity-Free Deferral (IFD) removes such shortcuts through role-indexed classwise competence profiles, but its estimates are constant within each class and cannot capture instance-level expert specialization. We propose RACER---Role-Aligned Competence Estimation for Routing---a role-relative framework for estimating an unseen expert's competence from context. RACER estimates the posterior-predictive probability that the expert is correct on a query under each candidate class role, then combines these estimates with the model posterior to obtain the Bayes-relevant expert-correctness probability. Nonparametric and neural kernel-pooling estimators use candidate-role relations, shared aggregation, and symmetric summaries, excluding absolute class-identity channels. We prove coherent class-relabelling invariance, derive a Bayes-aligned deferral surrogate, and give a plug-in regret bound relating routing regret to classifier and competence-estimation error. On controlled synthetic benchmarks, including a PathMNIST histopathology context-scaling study with simulated experts, RACER benefits from additional context under hidden subtype dependence and gives the strongest aggregate performance on a separately sampled unseen-expert split in the CIFAR-100 synthetic experiments. On the radiologist and human--AI chest-radiography benchmarks (VinDr-CXR and CheXpert), the RACER family is competitive or best in budget-swept deferral, with calibration results varying across metrics and datasets.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include RACER, learning to defer, human-AI collaboration.
Source attribution: curated from arXiv cs.LG (@Joshua Strong, Emma Sun, Alexander Capstick, Pramit Saha, Cheng Ouyang, J. Alison Noble).
