Vision-Language Models Misjudge When a Failing Robot Should Ask Humans
When Should a Failing Robot Ask? Initiating Corrective Human-Robot Dialogue from Audited Sensor Evidence
Robotics teams choosing open vision-language models get hard numbers: force telemetry beats camera frames 0.99 to 0.55 for failure diagnosis, and answer ordering alone swings refusal rates from near-total to near-zero.
Researchers built a simulated robot-failure benchmark in which every failure cause is injected and known, measuring what each sensor reveals with explicit leakage checks: failure diagnosis reaches 0.
99 from force data while no image-based method exceeds 0.55.
Across six open vision-language models tested, moving the refusal option from last to first in the answer list collapsed refusal rates from 78-100% to 0-6% in three of six model-and-family pairs.
Accuracy from camera frames stayed at or below a majority-class baseline under every prompt variant, with or without worked examples, and stated confidence carried no information about correctness.
Feeding the same models force data as ten lines of text produced the first above-baseline diagnoses in four of the six models.
Framed as a three-action decision problem of acting, consulting own sensors, or asking a human, the models ignored a fourfold change in question cost, though a single question lifted diagnoses to the answerer's reliability of 0.
70-0.81.
When Should a Failing Robot Ask? Initiating Corrective Human-Robot Dialogue from Audited Sensor Evidence
A robot that fails at a task faces the first decision in corrective dialogue: act on its own diagnosis, consult another onboard sensor, or interrupt a person. Choosing well requires knowing how much the robot's sensors reveal about the cause and how reliable the robot's own diagnosis is. We build a simulated benchmark in which every failure's true cause is known, because we injected it, and measure what each sensor reveals, with explicit checks against data leakage. Some failures are diagnosable from camera images; others only from the robot's force data (0.99 from force data, no image method above 0.55). We then test six open vision-language models. Their behavior tracks the surface of the prompt, not the evidence: moving the refusal option from last to first in the answer list collapses refusal rates from 78-100% to 0-6% in three of the six swept model-and-family pairs. Accuracy from frames stays at or below a majority-class baseline under every prompt variant, with or without worked examples, and stated confidence carries no information about correctness. Handing the same models the force data as ten lines of text produces the first above-baseline diagnoses, in four of the six models: much of the failure reflects missing sensor data, not missing ability. We pose the choice as a three-action decision problem, act, consult your own sensors, or ask a human, whose optimal policy follows from measured accuracy. The models do not follow it, and their ask rates ignore a fourfold change in question cost. One question to a human still lifts them from that baseline to roughly the answerer's own reliability (0.70-0.81 when they ask). The decision to ask should be tied to measured accuracy and stated costs, not to the model's confidence.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include multimodal, open-source models, prompt engineering, human-computer interaction.
Source attribution: curated from arXiv cs.AI (@Eshika Pathak, Leela Krishna).
