Research map / Vision-language and multimodal / Cross-modal learning
Scaling Language-centric Omnimodal Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that text-only LoRA contrastive learning on MLLM decoders transfers to non-text modalities via latent cross-modal alignment, proposes LCO-Emb, and identifies a generation-representation scaling law with a PAC-Bayesian bound.
Problem
Why MLLM-based multimodal embedding models outperform CLIP-style models is underexplored; the paper asks whether MLLMs have latent cross-modal alignment from generative pretraining and how representation quality relates to generative ability.
Approach
Treat contrastive learning as a lightweight post-hoc refinement on a language decoder with LoRA using only text pairs (all-NLI, Scale-1M); the alignment learned during generative pretraining transfers to image, audio and video embeddings. Also proposes GRSL, a positive scaling between MLLM generative ability and post-CL representation quality, with a PAC-Bayes bound.
Key results
- LCO-Emb-Omni (7B, multimodal, 370k) achieves 67.6 mean on MIEB-Lite English and 68.8 multilingual, setting a new state of the art, with ~21x less data than GME (~8M pairs).
- Text-only LCO-Emb-VL (7B) outperforms E5-V and Voyage-Multimodal-3 on MIEB-Sub18 by 21.69 and 13.00 points on average.
- LoRA text-only training (avg 71.98 on Qwen2.5-VL-7B) beats CLIP-style multimodal CL (50.02) at ~50x lower training time, and full fine-tuning (66.49).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LCO-Emb-Omni 7B multimodal instead of the strongest reported baseline | MIEB-Lite embedding models · MIEB-Lite (51 tasks) | Mean (m) | Helped |
| Add approximately 94k multimodal pairs to the text-only training data | LCO-Emb-VL · MIEB-Lite (51 tasks) | Mean (m) | Helped |
| Use LoRA instead of full fine-tuning (ablation) | Qwen2.5-VL-3B · MIEB-Sub18 | Average: 62.10; 66.28 | Helped |
| Use LoRA instead of full fine-tuning (ablation) | Qwen2.5-VL-7B · MIEB-Sub18 | Average: 66.49; 71.98 | Helped |
| Replace text-only LoRA with CLIP-style multimodal contrastive learning (ablation) | Qwen2.5-VL-3B · MIEB-Sub18 | Average: 66.28; 53.38 | Hurt |
| Replace text-only LoRA with a linear projection (ablation) | Qwen2.5-VL-3B · MIEB-Sub18 | Average: 66.28; 52.69 | Hurt |
| Combine the all-NLI and Scale-1M fine-tuned variants by model soup (ablation) | LCO-Emb · MIEB-Sub18 | Average: 71.17; 71.11; 72.17 | Helped |
| Use all-NLI rather than Scale-1M training data (ablation) | LCO-Emb · MIEB-Sub18 Visual STS English | Spearman correlation: 84.69; 81.27 | Helped |
| Use Scale-1M rather than all-NLI training data (ablation) | LCO-Emb · MIEB-Sub18 Linear Probing | accuracy: 58.61; 51.86 | Helped |
| Use Scale-1M rather than all-NLI training data (ablation) | LCO-Emb · MIEB-Sub18 Multilingual Image Retrieval | nDCG@10: 64.81; 56.37 | Helped |
| Increase LoRA alpha substantially for out-of-distribution tasks (ablation) | LCO-Emb · Compositionality and vision-centric QA | performance | Hurt |
| Set LoRA rank to 256 and alpha to 512 (ablation) | LCO-Emb · MIEB-Sub18 | Average: 67.70; 19.31* | Hurt |
Surprising findings
- Text-only contrastive learning on the language decoder reduces anisotropy and improves alignment for image, audio and video embeddings, despite no non-text supervision.
- CLIP-style multimodal contrastive fine-tuning underperforms text-only LoRA and can even degrade some tasks (e.g., multilingual image retrieval drops for Qwen2.5-VL-3B).
Limitations
- Joint generative and contrastive training was not explored due to computational cost.
- LoRA rank and alpha have no global optimum; large alpha degrades OOD tasks such as compositionality and VQA.
- The theoretical GRSL bound relies on approximations (I_P ≈ H(Y) − L_g and a hypothesis about warm start) that are not formally proven.
Experimental setup
Datasets: MNLI SNLI Scale-1M (20M multilingual parallel corpora subset) PixmoCap AudioCaps MSR-VTT ColPali Docmatix MS-COCO Clotho ActivityNet ViDoRe
Benchmarks: MIEB-Lite MIEB-Sub18 AudioCaps Clotho MSR-VTT ActivityNet SeaDoc TextVQA DocVQA OCRBench ChartQA Video-MME
Metrics: nDCG@10 nDCG@5 Recall@1 accuracy Spearman correlation NMI anisotropy mutual kNN
Baselines: CLIP-ViT-bigG SigLIP-so400m VLM2Vec E5-V Voyage Multimodal 3 mmE5 GME
Builds on: E5-V CLIP LoRA model soup MIEB
Closest papers by meaning
- Incorporating Dense Knowledge Alignment into Unified Multimodal Representation Models (CVPR 2025 · similarity 0.90)
- UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddings (ICLR 2026 · similarity 0.89)
- VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks (ICLR 2025 · similarity 0.89)
- NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints (NeurIPS 2025 · similarity 0.89)
- Unified Generative and Discriminative Training for Multi-modal Large Language Models (NeurIPS 2024 · similarity 0.89)
- AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.