这篇论文找到了一种聪明的方法:先快速扫一遍图像找出可能的目标区域,再只对这几个区域做多模态融合,省了很多计算。适合想做轻量级多模态目标检测的人读。
本文提出一种稀疏跨模态融合机制用于RGB-T目标检测,避免传统方法中双重骨干网络和全局融合的高计算成本。该方法先通过轻量级单模态检测器快速扫描图像,生成高召回率的候选区域(RoI),再对稀疏的候选区域进行跨模态特征融合以精化检测结果。两阶段框架显著降低了参数和计算成本,同时在高分辨率图像上保持可扩展性。实验证明该方法在保持竞争力的前提下实现高效检测。
Efficient RGB-T Object Detection via Sparse Cross-Modality Fusion
RGB-T detectors leverage the complementary strengths of visible and thermal infrared modalities, achieving robust performance under challenging conditions. Many of them resort to heavy dual backbones and exhaustive cross-modality fusion across the entire image, leading to impractically high computational costs. We observe that most image regions are smooth backgrounds (e.g., sky, ground) that can be easily handled by lightweight single-modality models. In light of this observation, we propose a sparse fusion mechanism for efficient RGB-T detection: first rapidly scanning the image to identify the proposals and then carefully examining the remaining sparse proposals via feature fusion. We propose a two-stage framework to instantiate this mechanism, which performs detection in two stages: 1) a lightweight and modality-specific detection stage that produces high-recall RoIs, and 2) a fusion-driven examination and refinement stage that filters out the false positives and refines the bounding boxes. This design enables the detector to adaptively allocate more computational resources to the potential foregrounds, improving the efficiency while ensuring detection accuracy. Extensive experiments show that our method achieves competitive performance with substantially fewer parameters and lower cost, while maintaining strong scalability to high-resolution images.