Research map / Vision-language and multimodal / Cross-modal learning

M3-JEPA: Multimodal Alignment via Multi-gate MoE based on the Joint-Embedding Predictive Architecture

ICML 2025 · Vision-language and multimodal › Cross-modal learning · approach: Multimodal fusion with missing modalities · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes M3-JEPA, which applies a Joint-Embedding Predictive Architecture with a multi-gate mixture-of-experts predictor and alternating gradient descent to align pretrained unimodal encoders' embeddings across vision-language, audio-language, and classification tasks.

Problem

Cross-modal alignment built on token-space optimization in multimodal LLM-style systems suffers from modality collapse and gradient conflicts, and existing approaches lack a generalized any-to-any latent alignment framework.

Approach

Use a JEPA-style latent predictor implemented as a multi-gate MoE (one gate per loss: contrastive and regularization) that maps input embeddings to output embedding space, with modality-specific and shared gating, trained by alternating gradient descent across directional tasks.

Key results

• On COCO image-to-text retrieval, M3-JEPA reaches R@1 87.7 with 140M trainable parameters, versus 84.8 for BEiT-3 and 85.4 for BLIP-2 ViT-g (1.2B).
• On ImageNet-1K, M3-JEPA reports 86.6% accuracy versus 83.2% for DINOv2 and 82.1% for CLIP-ViT.
• On VQAv2 test-dev, M3-JEPA scores 82.3, second to BEiT-3 at 84.2; the ablation shows removing AGD drops COCO image-to-text R@1 from 87.7 to 68.2 with MoE only.

Measured effects reported in the paper

Surprising findings

• Simple concatenation of image and question embeddings for VQA trails BEiT-3 and the authors attribute failures to lack of fine-grained cross-modal fusion.
• Training with a single shared MoE and no alternating optimization degrades COCO R@1 far more than removing MoE, per the ablation.

Limitations

• JEPA is not generative, so it cannot directly produce text or other outputs.
• Adding a new modality requires manually selecting an encoder and adding a modality-specific expert; not modality-agnostic.
• Multimodal input via simple concatenation underperforms on VQA (e.g., misses fine details such as stairs).

Experimental setup

Datasets: COCO Flickr30K Clotho AudioCaps WavText5K Freesound ImageNet-1K VQAv2 NLVR-2

Benchmarks: COCO Flickr30K Clotho AudioCaps ImageNet-1K VQAv2 NLVR-2

Metrics: R@1 R@5 R@10 accuracy precision recall F1 score VQA score

Baselines: CLIP ALIGN FILIP Florence BEiT-3 BLIP-2 LanguageBind DinoV2 CLIP-ViT BLIP

Builds on: JEPA multi-gate mixture-of-experts LanguageBind COMPLETER

Closest papers by meaning

• Gramian Multimodal Representation Learning and Alignment (ICLR 2025 · similarity 0.87)
• AlignMamba: Enhancing Multimodal Mamba with Local and Global Cross-modal Alignment (CVPR 2025 · similarity 0.86)
• Aligning Multimodal Representations through an Information Bottleneck (ICML 2025 · similarity 0.86)
• Mind the Heads: Topological Representation Alignment for Multimodal LLMs (NeurIPS 2026 · similarity 0.86)
• Guiding Cross-Modal Representations with MLLM Priors via Preference Alignment (NeurIPS 2025 · similarity 0.86)
• Modality-Aware SAM: Sharpness-Aware-Minimization Driven Gradient Modulation for Harmonized Multimodal Learning (NeurIPS 2025 · similarity 0.86)