MODUS: 仅解码器的任意到任意多模态建模
MODUS: Decoder-Only Any-to-Any Modeling of Diverse Modalities
MODUS用一个解码器统一处理各种模态的输入输出，省去多个专用组件，性能不输专家模型，还开源了。
MODUS是一个仅解码器的any-to-any多模态模型，无需模态特定的输出头或损失函数，就能从任意组合的输入模态预测任意输出模态。
它在视频、图像、语言等模态上直接使用预训练解码器，无需重头训练。在多个基准测试中，MODUS与专业模型和多任务基线性能相当甚至更优。
所有代码和模型权重已在 modus-multimodal.epfl.
ch 开源。
MODUS: Decoder-Only Any-to-Any Modeling of Diverse Modalities
Any-to-any models predict any modality from any combination of others within a single network, a formulation used in multimodal vision and vision-language models, and increasingly in scientific domains such as ecology and astronomy. Existing any-to-any models are typically trained from scratch using encoder-decoder or diffusion architectures, impacting their performance and preventing them from using strong pre-trained decoder-only models as a prior. In this work, we investigate decoder-only any-to-any multimodal modeling, which treats all modalities symmetrically and supports arbitrary modalities as inputs and outputs without modality-specific heads, losses, or task pipelines. Because every modality is both an input and an output of the same model, the resulting model, named Modus, can support a range of applications, such as chained generation through intermediate modalities or cross-modal self-verification by scoring the model's own outputs with another generated modality. Modus demonstrates strong out-of-the-box performance and is competitive with specialist and multitask baselines using a single model across various benchmarks. All materials are open-sourced at https://modus-multimodal.epfl.ch/.
This report is categorized under AITOP's "Models" desk. Key entities and technical topics under continuous tracking include MODUS, 多模态, 解码器架构, any-to-any.
Source attribution: curated from arXiv cs.AI (@Mingqiao Ye, Zhaochong An, Zhitong Gao, Xian Liu, François Fleuret, Chuan Li, Amir Zadeh, Serge Belongie, Afshin Dehghan, Jesse Allardice, David Mizrahi, Oğuzhan Fatih Kar, Roman Bachmann, Amir Zamir).
