MAP基准测试首次评估AI系统如何帮助有无障碍需求的用户规划实地访问，包含主张验证和视觉证据检索两项创新评估。
MAP是首个评估多模态AI系统辅助无障碍需求用户规划实地访问的基准。该基准包含两项新颖评估：无障碍规划主张验证，评估系统对场所信息和无障碍功能信息的支持能力；无障碍规划视觉证据检索，检查系统能否为请求的场所和无障碍功能选择视觉证据。MAP采用自动评分和部分人工评分相结合的方法，支持在场所信息和无障碍信息随时间变化的环境下比较AI系统性能。
MAP: A Benchmark on Multimodal Accessibility Planning for Real World Places
We introduce MAP, the first benchmark to evaluate multimodal AI systems as assistants for users with accessibility requirements when planning visits to places in the real world. In our evaluation, systems are presented with requests to verify or recommend a point of interest meeting an accessibility requirement. MAP contains two novel assessments: Claim verification for accessibility planning assesses if information on places and stated accessibility features is supported and identifies places that satisfy requested accessibility features. Visual evidence retrieval for accessibility planning checks if a multimodal AI system can select visual evidence for the requested place and accessibility feature. Our methodology supports comparison of AI systems in a setting where place information and accessibility information can change over time by evaluating systems and refreshing ground truth data at scheduled times. The benchmark is based on automatic rating and human rating for a proportion of responses.