Research map / Vision-language and multimodal / CLIP and zero-shot

MMRL: Multi-Modal Representation Learning for Vision-Language Models

CVPR 2025 · Vision-language and multimodal › CLIP and zero-shot · approach: Test-time adaptation of VLMs · 22 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MMRL, a shared learnable modality-agnostic representation space whose tokens are projected into higher layers of CLIP's image and text encoders for few-shot adaptation, with a decoupled inference strategy that preserves generalization.

Problem

Adapting pre-trained vision-language models such as CLIP with few-shot data leads to overfitting and loss of zero-shot generalization to new classes and datasets.

Approach

Introduce a shared, learnable representation space projected into both modalities' higher encoder layers as extra tokens, train representation features with a trainable projection while keeping the class-token projection frozen and regularized toward frozen CLIP features, and at inference use both feature types for base classes but only class features for novel classes.

Key results

• Base-to-novel average over 11 datasets: MMRL reaches 85.68 base, 77.16 novel, 81.20 HM, beating MMA (83.20/76.80/79.87) by 2.48, 0.36 and 1.33 points.
• Cross-dataset evaluation trained on ImageNet: MMRL averages 68.27 on the 10 target datasets... with ImageNet source accuracy 72.03 versus 71.00 for MMA, and the highest average over targets (reported 1.03% improvement over MMA).
• Few-shot learning across 11 datasets: MMRL gives the best average accuracy at every shot count from 1 to 16.

Measured effects reported in the paper

Surprising findings

• Inserting representation tokens into lower encoder layers degrades base-class performance, supporting the claim that higher layers hold dataset-specific features.

Limitations

• Novel-class accuracy does not win consistently on several datasets such as UCF101, EuroSAT, DTD and FGVCAircraft.
• Performance drops when the representation space or number of tokens is too large, indicating overfitting risk.

Experimental setup

Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT ImageNetV2

Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT ImageNetV2

Metrics: base accuracy novel accuracy harmonic mean top-1 accuracy

Baselines: CLIP CoOp CoCoOp ProDA KgCoOp MaPLe PromptSRC ProVP MetaPrompt TCP MMA

Builds on: CoOp MaPLe CLIP-Adapter PromptSRC MMA

Closest papers by meaning

• MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning (ICLR 2024 · similarity 0.92)
• MMA: Multi-Modal Adapter for Vision-Language Models (CVPR 2024 · similarity 0.91)
• VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks (ICLR 2025 · similarity 0.90)
• Towards General Continuous Memory for Vision-Language Models (NeurIPS 2025 · similarity 0.90)
• Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.89)
• Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.89)