HomeLoop：基于 Isaac Sim 的家庭机器人-IoT 协同仿真基准发布
HomeLoop: A Simulation Framework and Benchmark for Robot-IoT Coordination in Everyday Domestic Assistance and Safety
家庭机器人光聪明还不够，这个新基准专测它会不会瞎操作——六款 VLM 最高才 61.9 分，做机器人研究的可以关注。
论文提出 HomeLoop，一个构建在 Isaac Sim 上的仿真框架与基准，用于测试 VLM 驱动的家庭机器人与摄像头、IoT 设备、可穿戴设备的协同能力。基准包含 20 个任务、五大任务族，同一警报可能对应多种成因，要求机器人先推理再行动。评测六款开源与闭源 VLM，最高分仅 61.9%，且失败案例中过度干预（如未排查原因就关炉灶）数量是漏掉必要干预的两倍以上。框架将附带 2,100 种户型布局、5 种机器人本体和 20 类联网设备。
HomeLoop: A Simulation Framework and Benchmark for Robot-IoT Coordination in Everyday Domestic Assistance and Safety
Recent progress in embodied robotics is bringing robots closer to deployment in homes, as vision-language models (VLMs) are demonstrating significant advances in reasoning, making VLM-driven robots promising home assistant agents. These agents will often be introduced into homes that already have cameras, Internet of Things (IoT) devices, and residents with wearables. To fulfill their potential, they need to integrate with existing smart infrastructure and reason through ambiguous scenarios: a smoke alarm may come from a pan about to ignite, a pan already on fire, a resident cooking or a faulty sensor, and each calls for a different response. To that end, we introduce HomeLoop, a simulation framework and benchmark built on Isaac Sim that combines robot bodies, networked devices and wearables, simulated residents and visitors, faulty sensors, and hazards that evolve over time. Our benchmark is composed of 20 tasks across five families grounded in real-world household hazards and needs, and designed so that the same alert can have different causes. A runtime monitor scores each episode with temporal-logic checks on the final outcome and the actions along the way, and every action carries a predefined cost. We evaluate six open-weight and proprietary VLMs. The best achieves a family-balanced score of 61.9%, and across models we see a pattern of over-reaction: failed episodes involving prohibited interventions outnumber those involving missed required interventions by more than two to one, with models often shutting off the stove before investigating the alarm's cause. HomeLoop is an extensible foundation intended for use in research on robot-IoT coordination, long-term memory, proactive assistance, and privacy-safety trade-offs. HomeLoop will be released with 2,100 home layouts that can be automatically furnished in diverse configurations, five robot bodies and 20 networked device types.