The non-obvious takeaway is that the standard pipeline of pre-training a vision encoder separately may be unnecessary compute spending, and that where vision parameters sit inside the model matters more than having them at all. Teams training small multimodal models, especially edge or latency-sensitive deployments, get a concrete recipe plus faster first-token response. The 3x compute penalty for parameter sharing is a quantified warning against over-merging modalities.
#vision encoders
6 stories · 3 in 7 days · 3 in 30 days
Stories, products and papers tagged "vision encoders", newest first.
Nov 16
Meta and NYU Research Redesigns Vision Modules in Multimodal Models
Nov 13
Chinese content研究拆解多模态视觉编码三大设计选择:视觉先验并非必需
做视觉模型的开发者该看看这篇。它用统一数据和算力做对照实验,证明从头联合训练编码器不输预训练方案,视觉模块放进骨干前几层还能降低首 token 延迟,能直接指导你砍掉一条训练管线。
Chinese contentPoison-then-Hide:在预训练视觉编码器中埋入休眠后门的微调激活攻击
做视觉应用经常直接下开源编码器微调的朋友该看看。这篇论文证明被污染的编码器即使微调后后门也能存活,防御手段基本失效,最高 97% 攻击成功率,比已有的微调激活攻击适用面更广。
Sep 9
Jul 28
Models12:30
Chinese contentClinFusion: 面向整体医学理解的视觉中心多模态LLM系统ClinFusion用新架构统一2D/3D医学图像,在20/24基准上碾压Hulu-Med、Lingshu,甚至超GPT-5.2。医生盲评说它报告最好,值得关注。
Jul 21
Chinese content小鹏发布TuringViT视觉编码器,称数据效率提升10倍
小鹏自己搞了个TuringViT视觉编码器,只用了别人十分之一的数据就达到更好效果,还能跑车载和机器人,效率很高。
IT之家Source