Papers

FRAAC:面向法语象形图辅助通信的多模态大模型与数据集

FRAAC: A French Multimodal Large Language Model and Dataset forPictogram-Based Augmentative and Alternative Communication

Why we picked it

研究团队开源了 30 万法语象形图平行语料,还训了一个能看象形图、也能用象形图回答的模型。做无障碍、辅助通信或多模态方向的人可以直接用这批数据和 RAPS 标注思路,做双语和多模态研究的也能借鉴。

法国研究团队发布 FRAAC,针对多模态大模型无法可靠解读和生成象形图序列的问题,提出 RAPS(Retrieval-Augmented Pictogram Selection)标注方法。该方法用 LLM 概念抽取和象形图词表检索,修正规则转录系统在命名实体、数字、多义词等场景下的失败案例,无需编写新规则。应用于法语后,RAPS 生成了 140 万平行语料对,其中 30 万已公开发布。

Body

在人类排序研究中,标注员对同一句话的四种象形图转录进行排序,RAPS 超越了基于规则的最先进系统。研究还发现标准参考指标能区分质量差距明显的系统,但无法在强系统之间排出高低。团队最终构建了一个指令微调模型,支持多种 AAC 模态交互:接受象形图或文本输入,以象形图、语音或文本输出回答。

这项工作填补了辅助通信(AAC)领域文本-象形图平行数据稀缺和语法标准缺失的空白,为失语症等需要象形图交流的群体提供了可用的开源资源。

Original · arXiv cs.AI

FRAAC: A French Multimodal Large Language Model and Dataset forPictogram-Based Augmentative and Alternative Communication

Pictograms are a core modality in augmentative and alternative communication (AAC), yet recent multimodal large language models (MLLMs) cannot reliably interpret or produce pictogram sequences. Two obstacles explain this gap: the absence of a standardized pictogram grammar and the scarcity of parallel text-pictogram data. We introduce Retrieval-Augmented Pictogram Selection (RAPS), an annotation method that refines a rule-based transcription system's output with LLM-based concept extraction and retrieval over a pictogram vocabulary, handling cases where hand-written rules fail (named entities, numbers, polysemous words) without requiring new rules. Applied to French, RAPS yields a synthetic corpus of 1.4 million parallel pairs, of which 300k are publicly released. In a human ranking study where annotators order four pictogram transcriptions of the same sentence, RAPS outperforms the rule-based state of the art. Moreover, we show that standard reference-based metrics separate systems of clearly different quality but fail to rank strong systems against each other. Finally, we build an instruction-tuned model that supports interaction in several AAC modalities: it accepts pictogram or text input and answers in pictograms, speech, or text.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk, focusing on Europe's AI governance and technology ecosystem. Key entities and technical topics under continuous tracking include FRAAC, RAPS, multimodal.

Source attribution: curated from arXiv cs.AI (@Alexandre Audibert, François Portet, Benoit Favre, Maja Jonck Hjuler).