DeepSeek-OCR这类压缩方法旧评测没测准，ZeroSense基准去掉语义干扰后，压缩质量和下游准确率明显脱节。
以DeepSeek-OCR为代表的视觉-文本压缩方法在长上下文任务中宣称了很高的token压缩比。但现有评测依赖下游任务表现，MLLM的语义先验会掩盖文本保留的真实质量。新框架将视觉与语义能力解耦，ZeroSense Benchmark特意降低测试样本的语义关联。跨多个数据集的实验显示，压缩质量和下游准确率出现明显分歧，说明解耦式评测是必要的。
Decoupling semantics from vision: A framework for faithful visual-text compression evaluation
Recent visual-text compression (VTC) methods, typified by DeepSeek-OCR, report impressive high token compression ratios for long-context modeling tasks by leveraging text-to-image rendering. However, existing evaluation protocols heavily rely on downstream task performance. Such evaluation metrics fail to accurately measure text preservation due to the strong inherent linguistic priors of Multimodal Large Language Models (MLLMs). In this work, we introduce a new evaluation framework that decouples MLLMs' capabilities to faithfully assess VTC quality. Within this framework, we further introduce the ZeroSense Benchmark to ensure low semantic correlation of testing samples. By eliminating textual dependencies, our benchmark guarantees that the evaluation results are purely reflective of VTC quality, unaffected by the semantic inference capabilities of downstream models. Extensive experiments across multiple datasets demonstrate that VTC quality and downstream task accuracy diverge significantly, highlighting the necessity of our decoupled evaluation framework.