Apple Introduces Tri-Modal Masked Diffusion Models
The Design Space of Tri-Modal Masked Diffusion Models
Pretraining diffusion models across three modalities from scratch departs from the dominant recipe of finetuning a unimodal checkpoint, potentially simplifying how text, image and audio generation are unified in one architecture. Researchers and engineers working on discrete diffusion or multimodal models can reuse its scaling-law findings and sampling defaults. Apple's publication puts it in direct conversation with autoregressive approaches from labs like OpenAI and Google.
Apple researchers present the first tri-modal Masked Diffusion Models (MDM) pretrained from scratch on text, image-text and audio-text data, rather than finetuning a unimodal base model. The paper systematically analyses multimodal scaling laws, modality mixing ratios, noise schedules and batch-size effects. Its batch-size analysis produces a new stochastic differential equation (SDE)-based result, and the authors provide optimised inference sampling defaults for discrete diffusion.
The Design Space of Tri-Modal Masked Diffusion Models
Discrete diffusion models have emerged as strong alternatives to autoregressive language models, with recent work initializing and finetuning a base unimodal model for bi-modal generation. Diverging from previous approaches, we introduce the first tri-modal Masked Diffusion Models (MDM) pretrained from scratch on text, image-text, and audio-text data. We systematically analyze multimodal scaling laws, modality mixing ratios, noise schedules, batch-size effects and provide optimized inference sampling defaults. Our batch-size analysis yields a novel stochastic differential equation (SDE) based…
