清华团队推出PACE框架，让Qwen2.5-VL-7B推理速度提升3.1倍，性能只损失6.2%
PACE框架通过统一压缩提取范式，在Qwen2.5-VL-7B模型上实现仅使用10%视觉 tokens 保持93.8%原始性能，时间到首个token提速3.1倍。该框架包含自适应像素压缩器(APC)和动态双注意力提取器(DDAE)两个核心组件。PACE无需训练即可同时优化视觉编码器和大型语言模型的推理过程。
PACE: A Unified Condense-and-Extract Paradigm for Fast VLM Inference
Vision-Language Models (VLMs) demonstrate exceptional visual reasoning capabilities, yet their inference costs escalate rapidly with the proliferation of visual tokens. Existing visual token pruning methods exhibit two fundamental limitations. First, most approaches operate exclusively post-vision encoder, leaving the substantial latency of the visual encoding phase unoptimized. Second, under strict token budgets, these methods often fail to jointly preserve holistic visual contexts and fine-grained details, leading to performance degradation. To address these bottlenecks, we propose PACE (Pixel-Adaptive Condense and Extract), a training-free inference framework that accelerates both the vision encoder and the Large Language Model (LLM) via a unified Condense-and-Extract paradigm. During the Condense stage, an Adaptive Pixel Compressor (APC) evaluates visual information density prior to encoding, adaptively downsampling redundant inputs, curtailing encoder computation while preserving global context and essential visual cues. In the Extract stage, a Dynamic Dual-Attention Extractor (DDAE) selectively retains visual tokens via a fusion of internal visual signals from the encoder and semantic signals from the LLM, safeguarding task-critical details. By integrating PACE into Qwen2.5-VL-7B, the model retains 93.8% of its original performance while utilizing only 10% of the visual tokens, yielding a 3.1x speedup in time to first token (TTFT). Our code is available at https://github.com/jjL357/PACE.