#vision encoders

6 stories · 3 in 7 days · 3 in 30 days
Stories, products and papers tagged "vision encoders", newest first.
Nov 16
PapersBuildersSelected00:13
Meta and NYU Research Redesigns Vision Modules in Multimodal Models

The non-obvious takeaway is that the standard pipeline of pre-training a vision encoder separately may be unnecessary compute spending, and that where vision parameters sit inside the model matters more than having them at all. Teams training small multimodal models, especially edge or latency-sensitive deployments, get a concrete recipe plus faster first-token response. The 3x compute penalty for parameter sharing is a quantified warning against over-merging modalities.

Gorden Sun@Gorden_SunSource
Nov 13
Chinese content研究拆解多模态视觉编码三大设计选择:视觉先验并非必需

做视觉模型的开发者该看看这篇。它用统一数据和算力做对照实验,证明从头联合训练编码器不输预训练方案,视觉模块放进骨干前几层还能降低首 token 延迟,能直接指导你砍掉一条训练管线。

arXiv cs.LG@Ellis Brown et al. (12)Source
Chinese contentPoison-then-Hide:在预训练视觉编码器中埋入休眠后门的微调激活攻击

做视觉应用经常直接下开源编码器微调的朋友该看看。这篇论文证明被污染的编码器即使微调后后门也能存活,防御手段基本失效,最高 97% 攻击成功率,比已有的微调激活攻击适用面更广。

arXiv cs.LG@Qixuan Jin et al. (5)Source
Sep 9
Jul 28
Jul 21