TGAD基准测试揭示:文本引导异常检测模型实际并未真正理解语言指令
A Structured Benchmark for Text-Guided Anomaly Detection: When Language Stops Conditioning the Decision
做工业视觉检测的团队会发现,当前号称支持文本引导的模型其实并不听指令——TGAD基准直接戳破了这个泡沫,建议点开看看你的模型是否真的被语言控制。
工业异常检测领域近年引入多模态视觉语言模型,声称可通过文本指令实现零样本或少样本检测。但现有评估方法沿用单模态基准,无法验证模型是否真正依赖文本条件。
本文提出TGAD结构化基准,通过三个递进场景测试:MVTec AD上的提示敏感性测试、组件级标注扩展、以及新构建的组装面板数据集APD。测试发现,三种代表性模型(生成式大视觉语言模型、无训练判别式、嵌入自适应判别式)均仅表面响应文本指令:移除物体名词后生成模型I-AUROC从97.
4降至82.6;
组件指令无法约束决策;在APD上图像级判别甚至低于随机水平(最低31.
5)。结果表明当前多模态异常检测系统的文本引导能力被高估,需要此类基准才能实现工业部署所需的可靠语言控制。
A Structured Benchmark for Text-Guided Anomaly Detection: When Language Stops Conditioning the Decision
Industrial anomaly detection has historically been a unimodal task. Recent multimodal vision-language models have produced systems that admit textual input alongside the image and are presented as enabling text-guided zero- and few-shot inspection. Yet these methods are evaluated with protocols inherited from unimodal benchmarks that hold the textual condition constant and therefore cannot measure whether language conditions the decision; whether reported gains reflect text guidance or strong pretrained visual features remains open. We introduce Text-Guided Anomaly Detection (TGAD), a structured benchmark that progressively increases the functional role of language across three scenarios: a controlled prompt-sensitivity setting on MVTec AD; a component-tagged extension of MVTec AD that requires the model to restrict its assessment to an instructed part; and the new Assembled Panel Dataset (APD), a realistic industrial setting that requires both defect-type and component-location knowledge. We evaluate one representative model per paradigm: generative large vision-language, training-free discriminative, and embedding-adaptive discriminative. In all three, the textual interface conditions the decision only superficially: prompt content is absorbed unless the object noun is removed (the generative model's I-AUROC drops from 97.4 to 82.6); component-level instructions do not constrain the decision once defects outside the instructed part are admitted as normal (from 90.3 to 66.3); and when both combine on APD, image-level discrimination collapses below the MVTec level, in one case below chance (71.2, 50.5, 31.5). These results suggest that standard benchmarks overstate the text-guided capabilities of current multimodal anomaly detection systems, and that a protocol of this kind is a prerequisite for models that can be reliably controlled through language for industrial deployment.
This report is categorized under AITOP's "Papers" desk. Key entities and technical topics under continuous tracking include anomaly detection, multimodal, benchmarks, vision-language models.
Source attribution: curated from arXiv cs.AI (@Stefano Samele, Eugenio Lomurno, Teodora Jovanovic, Sanjay Shivakumar Manohar, Alberto Crivellaro, Matteo Matteucci).