Research map / Vision-language and multimodal / Cross-modal learning
Incorporating Dense Knowledge Alignment into Unified Multimodal Representation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds DeKon5M, a contrastive dataset rich in dense visual-centric knowledge, and trains DeKR, a unified text/image/video/composed embedding model on Qwen2-VL that matches or beats CLIP-series models on cross-modal retrieval.
Problem
MLLM-based unified multimodal representation models lag far behind CLIP-series models on cross-modal retrieval, attributed to weak multimodal alignment from training data that lacks dense visual-centric knowledge.
Approach
Use dense image descriptions (many visual-centric knowledge points per sample) for contrastive training of a parameter-shared MLLM encoder, combined with video-text and composed retrieval data, and extend the same model to ranking via an instruction change.
Key results
- DeKR-7B averages 87.1 on zero-shot COCO-5K/Flickr image-text retrieval, 0.5 points above InternVL-C (86.6) and 3.2 above E5-V.
- DeKR-7B reaches 41.0 average on zero-shot video-text retrieval, 4.7 points above InternVideo-L (36.3).
- On zero-shot composed image retrieval (FashionIQ/CIRR), DeKR-7B averages 33.7 versus 31.8 for E5-V at 8B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with dense-knowledge DeKon image-text data instead of sparse-knowledge COYO data (ablation) | DeKR · COCO I2T | Recall: 48.68 -> 64.30 | Helped |
| add video-text retrieval data to image-text training data (ablation) | DeKR · COCO I2T | Recall: 64.30 -> 69.44 | Helped |
| add composed retrieval data to image-text and video-text training data (ablation) | DeKR · COCO I2T | Recall: 69.44 -> 70.48 | Helped |
| add video-text and composed retrieval data to image-text training data (ablation) | DeKR · COCO T2I | Recall: 33.62 -> 52.38 | Helped |
| use bidirectional rather than causal attention (ablation) | DeKR · COCO I2T | Recall: 69.26 -> 70.48 | Helped |
| use bidirectional rather than causal attention (ablation) | DeKR · MSR-VTT V2T | Recall: 43.00 -> 42.00 | Hurt |
| increase model size from 2B to 7B (ablation) | DeKR · zero-shot image-text retrieval | Avg.: +2.8 points | Helped |
| increase training data from 625K to 5M samples (ablation) | DeKR · image retrieval | retrieval performance | Helped |
| increase dense-knowledge proportion from 0 to 80% (ablation) | DeKR · image retrieval | retrieval performance | Helped |
| increase dense-knowledge proportion from 80% to 100% (ablation) | DeKR · image retrieval | retrieval performance | Hurt |
| use DeKR representation training before ranking-data training rather than training ranking data on Qwen2VL (ablation) | DeKR-Rank · COCO-5K T2I ranking | R1: 39.0 -> 60.2 | Helped |
| train with dense knowledge and multimodal tasks rather than sparse COYO-only data (ablation) | DeKR · COCO I2T | Recall: 48.68 -> 70.48 | Helped |
Surprising findings
- Qwen2-VL without any training already performs zero-shot image-text ranking via prompting, suggesting MLLMs have strong inherent image-text alignment.
- Using only 80%+ dense knowledge data causes a significant performance drop, attributed to length generalization issues, so short-text data must be mixed in.
Limitations
- Gains on video-text retrieval rely on general public video data without extensive optimization.
- Performance declines when training data is predominantly long dense captions, requiring a mix of short texts.
Experimental setup
Datasets: DeKon5M COYO SBU SAM LLaVA-Pretrain ALLaVA VISTA WebVid InternVid Flickr30k COCO
Benchmarks: COCO-5K Flickr30k MSR-VTT DiDeMo FashionIQ CIRR ReMuQ WebQA
Metrics: Recall@1 Recall@5 Recall@10
Baselines: CLIP-L OpenCLIP-bigG Eva02-E+ SigLip-so400M InternVL-C VISTA E5-V Pic2Word LinCIR InternVideo-L ViCLIP Qwen2VL-PE
Builds on: CLIP E5-V LLaVA-NeXT Ernie-VIL Rosita
Closest papers by meaning
- Scaling Language-centric Omnimodal Representation Learning (NeurIPS 2025 · similarity 0.90)
- Beyond Global Similarity: Multi-Conditional Retrieval for Fine-Grained Cross-Modal Understanding (CVPR 2026 · similarity 0.89)
- Bridging Modalities: Improving Universal Multimodal Retrieval by Multimodal Large Language Models (CVPR 2025 · similarity 0.89)
- MM-EMBED: UNIVERSAL MULTIMODAL RETRIEVAL WITH MULTIMODAL LLMS (ICLR 2025 · similarity 0.89)
- Guiding Cross-Modal Representations with MLLM Priors via Preference Alignment (NeurIPS 2025 · similarity 0.88)
- SAE-V: Interpreting Multimodal Models for Enhanced Alignment (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.