Research map / Vision-language and multimodal / CLIP and zero-shot
Unified Vision–Language Modeling via Concept Space Alignment
Yifu QIU, Paul-Ambroise Duquenne, Holger Schwenk
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| align Perception Encoder to SONAR (vSONAR) vs original PE | Perception Encoder · PE-Video retrieval | Recall@1: 63.91 -> 73.03 | Improved |
| align Perception Encoder to SONAR (vSONAR) vs original PE | Perception Encoder · DREAM-1k retrieval | Recall@1: 72.10 -> 63.30 | Hurt |
| use frozen linear projector instead of full PE fine-tuning (ablation) | vSONAR · PE-Video captioning | BLEU: 37.1 -> 38.0 | Improved |
| add asynchronous learning rates for projector and encoder (ablation) | vSONAR · PE-Video captioning | BLEU: 37.1 -> 39.7 | Improved |
| add temporal attention layer (ablation) | vSONAR · PE-Video captioning | BLEU: 39.8 -> 39.8 | No clear effect |
| add stage-1 image captioning (12M) data (ablation) | vSONAR · PE-Video captioning | BLEU: 39.8 -> 40.1 | Improved |
| add synthetic video captioning stage (ablation) | vSONAR · PE-Video captioning | BLEU: 39.6 -> 40.1 | Improved |
| add contrastive loss to MSE alignment (ablation) | vSONAR · PE-Video | retrieval vs captioning: R@1 49.0 -> 52.4; BLEU 38.9 -> 38.6 | Mixed |
| fine-tune LCM on vision-language instructions (v-LCM) | LCM · M3IT IVQA | ROUGE-L: 48.9 -> 63.9 | Improved |
| fine-tune LCM on vision-language instructions (v-LCM) | LCM · M3IT VCR | token F1: 0.385 -> 0.671 | Improved |
| use OmniSONAR instead of SONAR 1 as alignment target | vSONAR · video captioning | BLEU | Improved |
| feed vSONAR embeddings instead of SONAR text re-encoded captions to LCM | LCM · VideoXum | ROUGE-L | Improved |
Closest papers by meaning
- Bridging Vision and Language Spaces with Assignment Prediction (ICLR 2024 · similarity 0.91)
- VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks (ICLR 2025 · similarity 0.91)
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.90)
- Towards General Continuous Memory for Vision-Language Models (NeurIPS 2025 · similarity 0.89)
- Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.89)
- Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
