Research map / Vision-language and multimodal / Cross-modal learning

UNIALIGN: Scaling Multimodal Alignment within One Unified Model

CVPR 2025 · Vision-language and multimodal › Cross-modal learning · approach: Multimodal fusion with missing modalities · 8 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes UNIALIGN, a single modality-aware MoE+LoRA encoder initialized from CLIP that aligns six modalities in one training phase using unpaired cross-dataset class-label similarity and distillation from anchor modalities and partial-modal foundation models.

Problem

Multimodal alignment typically needs a separate encoder and separate training per extended modality, so parameters and training cost grow with modality count and cross-modal knowledge transfer is limited.

Approach

A unified modality-aware mixture-of-experts with LoRA experts on a frozen CLIP ViT-B, trained jointly on unpaired data from all datasets via soft modality binding (class-label similarity as soft targets) and knowledge distillation from anchor modalities and partial-modal teacher models.

Key results

• Achieves 85.3% top-1 on ImageNet-1K with 7.8M trainable parameters, comparable to Hiera's 85.2% with 70.0M.
• Reaches 71.4% top-1 on NYU-D depth scene classification, surpassing ImageBind (54.0) and VIT-LENS (68.5) in the table.
• Scores 97.4% on Speech Commands V2 versus 97.0% for MetaFormer (86.3M trainable), and 68.5% zero-shot on UCF101 outperforming ImageBind's 64.8.

Measured effects reported in the paper

Surprising findings

• Naively adding modalities with only the contrastive loss degrades ImageNet-1K-independent performance on depth and point cloud tasks (SUN-D 40.4 with 6 modalities vs 34.4 with 3 modalities on ablation rows, showing mixed effects of scaling).
• Aligning depth only to text yields much lower accuracy (46.3 vs 71.4 with image+text anchors on NYU-D), suggesting depth aligns more closely with images.

Limitations

• Performance trails specialized SOTA on several tasks (e.g., zero-shot 3D classification, audio classification, audio-text retrieval) partly due to less training data (0.5M AudioSet videos).
• Experiments limited to two experts per modality and top-2 routing due to GPU memory constraints.

Experimental setup

Datasets: ImageNet-1K ShapeNet SUN RGB-D AudioSet VGGSound ESC-50 AudioCaps MSR-VTT UCF101 ModelNet40 ScanObjectNN NYU-D

Benchmarks: ImageNet-1K ModelNet40 ScanObjectNN NYU-D SUN RGB-D AudioSet ESC-50 AudioCaps Speech Commands V2 MSR-VTT UCF101

Metrics: top-1 accuracy top-5 accuracy mAP Recall@k trainable parameters

Baselines: ImageBind VIT-LENS LanguageBind ULIP MetaFormer Hiera AudioCLIP UniBind

Builds on: CLIP ImageBind LoRA Mixture-of-Experts ULIP

Closest papers by meaning

• With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide You (NeurIPS 2025 · similarity 0.90)
• Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs (NeurIPS 2026 · similarity 0.90)
• DeepAlign: Mitigating Modality Conflict through Modality-Specific Alignment (CVPR 2026 · similarity 0.90)
• MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data (NeurIPS 2026 · similarity 0.89)
• OneLLM: One Framework to Align All Modalities with Language (CVPR 2024 · similarity 0.89)
• AlignMamba: Enhancing Multimodal Mamba with Local and Global Cross-modal Alignment (CVPR 2025 · similarity 0.89)