LUMIR：基于异构数据的解耦多模态表征学习框架，用于下一个 POI 推荐
LUMIR: Decoupled Multimodal Representation Learning from Heterogeneous Data for Next-POI Recommendation
搞推荐系统又想接多模态数据的朋友可以看看。它用 LLM 把图片等异构数据统一成文本表征再融合，表征训练和下游推荐模型解耦，换个骨干也能复用，和那些绑死单一模型的融合方案思路不一样。代码开源，可以直接跑。
LUMIR 是一个解耦的多模态表征学习框架，采用对比学习，目标是旅游场景下的下一个兴趣点（next-POI）推荐任务。它针对的问题是：现有方法把多模态融合与特定下游模型的架构和优化目标紧耦合，导致学到的表征难以复用。
技术上分三步：先用大语言模型（LLMs）把来自异构数据源的多模态信息语义统一到共享文本表征空间；再设计交互感知融合机制建模跨模态交互；同时把序列信号作为结构化约束引导表征学习。实验在一个多模态 POI 数据集上展开，横跨多个序列推荐骨干模型，在准确率和准确率之外的指标上都超过现有多模态融合基线。
论文的价值在于验证了“表征学习与下游优化解耦”作为可复用多模态表征的有效范式，代码已在 GitHub 公开（wzehui/LUMIR）。
LUMIR: Decoupled Multimodal Representation Learning from Heterogeneous Data for Next-POI Recommendation
As multimodal tourism-related information collected from heterogeneous data sources has become increasingly available, learning effective representations has attracted attention for improving recommendation performance. Existing approaches often tightly couple multimodal fusion with the architecture and optimization objective of a specific downstream model, which limits the generality and reusability of the learned representations. To address these limitations, we propose LUMIR, a decoupled multimodal representation learning framework based on contrastive learning. Multimodal information collected from heterogeneous data sources is first semantically unified into a shared textual representation space using large language models (LLMs). An interaction-aware fusion mechanism is then designed to model cross-modal interactions, while sequential signals are incorporated as a structural constraint to guide representation learning. To evaluate the learned representations, experiments are conducted on a multimodal point-of-interest (POI) dataset under the next-POI recommendation setting across multiple sequential recommendation backbones. The results show that LUMIR consistently outperforms existing multimodal fusion baselines in both accuracy-oriented and beyond-accuracy metrics, demonstrating that decoupling representation learning from downstream optimization provides an effective paradigm for learning reusable multimodal representations. The code is publicly available at https://github.com/wzehui/LUMIR.
This report is categorized under AITOP's "Papers" desk. Key entities and technical topics under continuous tracking include LUMIR, 多模态, 推荐系统, 对比学习.
Source attribution: curated from arXiv cs.LG (@Zehui Wang, Wolfram Höpken, Dietmar Jannach).