Research map / Vision-language and multimodal / Cross-modal learning
Guiding Cross-Modal Representations with MLLM Priors via Preference Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MAPLE, which uses an off-the-shelf MLLM as a reward model to score hard-negative image-caption candidates and trains an MLLM-based embedding retriever with a Relative Preference Alignment (DPO-derived) loss plus a contrastive regularizer.
Problem
CLIP-style embeddings exhibit a modality gap and coarse contrastive alignment that fails at fine-grained cross-modal distinctions, and MLLM-based retrievers lose the inherent alignment of their generative backbones during fine-tuning.
Approach
Use an off-the-shelf MLLM's yes/no logit-based alignment scores to build pairwise and listwise preferences over DINOv2-mined hard negatives and captions, then adapt the reference-free simplified DPO objective to embedding similarities (RPA loss), combined with a contrastive regularizer.
Key results
- MAPLE (Qwen2-VL-7B) reaches 56.0 text / 32.7 image on Winoground vs 40.5 / 17.5 for VladVA (LLaVA-1.5-7B), the best prior MLLM retriever in the table.
- On COCO, MAPLE (Qwen2-VL-7B) gets 75.5 text / 60.3 image R@1, above the best CLIP-based and MLLM-based baselines listed.
- Ablation: contrastive loss plus listwise RPA raises NaturalBench text from 61.4 to 69.2 and image from 62.5 to 71.2 over the contrastive-only baseline; Winoground image rises from 20.5 to 28.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MAPLE with Qwen2-VL-7B instead of the strongest reported baseline | Cross-modal retrieval · Winoground | Text score: 40.5 -> 56.0 | Helped |
| Add listwise RPA with contrastive regularization (ablation) | Qwen2-VL retriever · Winoground | Text score: 42.5 -> 51.0 | Helped |
| Add listwise RPA with contrastive regularization (ablation) | Qwen2-VL retriever · COCO | Text R@1: 74.0 -> 71.9 | Hurt |
| Use listwise RPA instead of pairwise RPA with contrastive regularization (ablation) | Qwen2-VL retriever · Winoground | Text score: 49.8 -> 51.0 | Helped |
| Use preference contrastive loss on preference examples without RPA (ablation) | Qwen2-VL retriever · COCO | Text R@1: 74.0 -> 64.6 | Hurt |
| Add expanded negative pool without RPA (ablation) | Qwen2-VL retriever · COCO | Text R@1: 74.0 -> 75.3 | Helped |
| Add expanded negative pool without RPA (ablation) | Qwen2-VL retriever · MMVP | Text score: 33.3 -> 39.3 | Helped |
| Add expanded negatives and listwise RPA (ablation) | Qwen2-VL retriever · Winoground | Text score: 42.5 -> 56.0 | Helped |
| Increase caption sampling diversity from strategy A to strategy C (ablation) | Contrastive + listwise RPA training · Winoground | Text score: 49.0 -> 51.0 | Helped |
| Increase the RPA weight relative to contrastive loss (ablation) | MAPLE · General versus fine-grained retrieval | Retrieval performance | Mixed |
| Apply listwise RPA instead of omitting listwise RPA (ablation) | MAPLE · MMVP | Delta gap: 19.15 -> 5.66 | Helped |
| Apply listwise RPA instead of omitting listwise RPA (ablation) | MAPLE · Winoground | Delta gap: 10.0 -> 7.22 | Helped |
Surprising findings
- Training with only the contrastive loss on preference-selected data degrades COCO retrieval and gives little fine-grained gain, while RPA losses alone yield large fine-grained gains but hurt general retrieval.
- Off-the-shelf Qwen2-VL-7B shows a larger discriminative gap than the trained retriever on Winoground under the proposed Wasserstein-based metric, though the trained model narrows the distributional gap.
Limitations
- Cross-modal representations may inherit biases from the underlying MLLM.
- Not validated on more complex tasks such as composed retrieval.
- Winoground modality-gap analysis uses only a random 100 of 400 instances due to compute cost of pairwise MLLM scoring.
Experimental setup
Datasets: OpenImage COCO Flickr30K Winoground NaturalBench MMVP BiVLC
Benchmarks: MS-COCO Flickr30K Winoground NaturalBench MMVP BiVLC
Metrics: Recall@1 Text score Image score modality gap Wasserstein distance
Baselines: CLIP OpenCLIP SigLIP SigLIPv2 EVA-CLIP E5-V VladVA
Builds on: DPO CPO PRO SemDeDup E5-V
Closest papers by meaning
- Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs (NeurIPS 2026 · similarity 0.91)
- DeepAlign: Mitigating Modality Conflict through Modality-Specific Alignment (CVPR 2026 · similarity 0.89)
- Scaling Language-centric Omnimodal Representation Learning (NeurIPS 2025 · similarity 0.89)
- SAE-V: Interpreting Multimodal Models for Enhanced Alignment (ICML 2025 · similarity 0.89)
- MergeMix: A Unified Augmentation Paradigm for Visual and Multi-Modal Understanding (ICLR 2026 · similarity 0.88)
- Incorporating Dense Knowledge Alignment into Unified Multimodal Representation Models (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.