MoI-Bench：面向移动端图像生成与编辑的基准发布
MoI-Bench: A Benchmark for Image Generation and Editing on Mobile Devices
一个专门测手机上跑的图像编辑模型的基准，1150 个样本、按操作和目标两个轴拆任务，比单一分数更能告诉你端侧模型差在哪。
MoI-Bench 是一个基于移动端用户需求构建的图像生成与编辑基准，包含 64 个子任务和 1150 个样本，覆盖 16 种操作（如移除、替换）和 8 类视觉目标（如人像、文字）。配套的 MoI-Eval 从指令遵循、视觉质量、细节保留三个维度设计任务专属评分标准，并用二维 Quality-Deployability 能力图替代单分数排名。评测显示当前端侧模型已能高效生成和编辑图像，但任务覆盖面比部署效率更需优先提升，其中文字替换能力和身份保留、文字渲染是主要短板。
MoI-Bench: A Benchmark for Image Generation and Editing on Mobile Devices
Developing on-device image models requires considering practical user needs, task types and device resource constraints. However, existing benchmarks provide limited guidance on identifying capability gaps between on-device models and server-side models. We introduce MoI-Bench, a benchmark grounded in mobile user needs, comprising 64 subtasks and 1150 samples. It defines 16 operations, such as removal and replacement, and eight visual target categories covering content such as portraits and text. These form two orthogonal axes whose applicable combinations define specific tasks, such as removing facial makeup. To better guide model improvement, we further propose MoI-Eval, which introduces task-specific criteria covering instruction following, visual quality and detail preservation. We further construct sample-level evaluation questions and aggregate their scores into overall quality and capability profiles along the operation, target and criteria axes. We evaluate representative on-device and server-side models and measure mobile deployment costs. Results show that this decoupled Operation x Target design provides finer-grained guidance for improving compact models. For example, it highlights the need to prioritize specific capability gaps (e.g., text-replacement) over broad application-level goals (e.g., improving text editing). Unlike conventional benchmarks that rank models using a single score, we introduce a two-dimensional Quality-Deployability capability map to assess models for practical mobile use. The map shows that current on-device models can achieve efficient generation or editing but expanding their task coverage deserves greater attention than further improving deployment efficiency. Further progress requires improving editing quality, particularly identity preservation and text rendering, while broadening task coverage.
