Research map / Vision-language and multimodal / Cross-modal learning
Dynamic Adapter with Semantics Disentangling for Cross-lingual Cross-modal Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DASD, a frozen-CLIP cross-lingual cross-modal retrieval method whose adapter weights are dynamically generated from semantics-disentangled caption features, achieving SOTA on Multi30K, MSCOCO and MSRVTT with few trainable parameters.
Problem
Cross-lingual cross-modal retrieval for low-resource languages without human-labeled target-language image/video-text pairs, where static adapters fail to handle varied target-language caption expressions.
Approach
Generate input-conditional adapter parameters from semantic-related and semantic-agnostic features of each target-language caption, disentangled via semantic consistency loss (L1 to source-language representation) and adversarial training.
Key results
- Under cross-lingual finetune on Multi30K/MSCOCO, DASD (134M trainable) reaches 88.5 DE/91.1 FR/87.6 CS on Multi30K and 90.0 ZH/89.1 JA on MSCOCO with MT data, outperforming CL2CM (e.g., 83.0 DE).
- In zero-shot on MSRVTT-based video retrieval, DASD sums to 162.7 versus 129.6 for MLA across eight languages.
- Plugging DASD into CCLM raises its SUM from 413.7 to 452.7 across five target languages.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace static adapters with input-conditional dynamic adapters (ablation) | DASD · Multi30K and MSCOCO, cross-lingual finetune | mAR: 88.3 -> 91.1 (FR); 87.0 -> 88.5 (DE); 83.1 -> 87.6 (CS); 85.3 -> 90.0 (ZH); 85.7 -> 89.1 (JA) | Helped |
| Add semantic-agnostic features to semantic-related features (ablation) | DASD dynamic parameter generation · Multi30K and MSCOCO, cross-lingual finetune | mAR: 90.2 -> 91.1 (FR); 87.7 -> 88.5 (DE); 86.8 -> 87.6 (CS); 89.1 -> 90.0 (ZH); 88.0 -> 89.1 (JA) | Helped |
| Add semantic-related features to semantic-agnostic features (ablation) | DASD dynamic parameter generation · Multi30K and MSCOCO, cross-lingual finetune | mAR: 89.9 -> 91.1 (FR); 87.8 -> 88.5 (DE); 86.3 -> 87.6 (CS); 88.7 -> 90.0 (ZH); 88.1 -> 89.1 (JA) | Helped |
| Add semantic consistency loss (ablation) | DASD · Multi30K and MSCOCO, cross-lingual finetune | mAR: 90.6 -> 91.1 (FR); 88.1 -> 88.5 (DE); 86.9 -> 87.6 (CS); 89.5 -> 90.0 (ZH); 88.5 -> 89.1 (JA) | Helped |
| Add adversarial loss (ablation) | DASD · Multi30K and MSCOCO, cross-lingual finetune | mAR: 90.7 -> 91.1 (FR); 87.9 -> 88.5 (DE); 87.3 -> 87.6 (CS); 89.6 -> 90.0 (ZH); 88.3 -> 89.1 (JA) | Helped |
| Add both semantic-consistency and adversarial losses (ablation) | DASD without either disentangling loss · Multi30K and MSCOCO, cross-lingual finetune | mAR: 90.1 -> 91.1 (FR); 87.4 -> 88.5 (DE); 86.5 -> 87.6 (CS); 89.1 -> 90.0 (ZH); 88.0 -> 89.1 (JA) | Helped |
| Remove the dynamic adapter from the 12th layer (ablation) | DASD · Multi30K and MSCOCO, cross-lingual finetune | mAR: 91.1 -> 90.4 (FR); 88.5 -> 87.3 (DE); 87.6 -> 86.8 (CS); 90.0 -> 89.4 (ZH); 89.1 -> 88.6 (JA) | Hurt |
| Increase dynamic-adapter hidden dimension from 16 to 32 (ablation) | DASD · Multi30K and MSCOCO, cross-lingual finetune | mAR: 90.2 -> 91.1 (FR); 87.9 -> 88.5 (DE); 87.1 -> 87.6 (CS); 89.6 -> 90.0 (ZH); 88.2 -> 89.1 (JA) | Helped |
| Increase dynamic-adapter hidden dimension from 32 to 64 (ablation) | DASD · Multi30K and MSCOCO, cross-lingual finetune | mAR: 91.1 -> 91.4 (FR); 88.5 -> 88.2 (DE); 87.6 -> 87.5 (CS); 90.0 -> 90.2 (ZH); 89.1 -> 89.3 (JA) | Mixed |
| Replace L1 semantic-consistency loss with L2 (ablation) | DASD · Multi30K and MSCOCO, cross-lingual finetune | mAR: 91.1 -> 89.5 (FR); 88.5 -> 88.2 (DE); 87.6 -> 87.1 (CS); 90.0 -> 89.4 (ZH); 89.1 -> 88.8 (JA) | Hurt |
| Increase training data from CC300K to CC3M (ablation) | DASD · Multi30K and MSCOCO, zero-shot | mAR: 88.6 -> 88.7 (FR); 87.4 -> 87.6 (DE); 83.4 -> 84.3 (CS); 88.5 -> 88.7 (ZH); 84.8 -> 85.3 (JA) | Helped |
| Increase target-language CLIP layers from 1 to 12 (ablation) | DASD · Multi30K and MSCOCO, cross-lingual finetune | mAR: 59.3 -> 91.1 (FR); 61.4 -> 88.5 (DE); 50.3 -> 87.6 (CS); 61.7 -> 90.0 (ZH); 59.7 -> 89.1 (JA) | Helped |
Surprising findings
- Using only the first CLIP layer as the backbone for semantics disentangling performs as well as using all 12 layers, and 0 pretrained layers gives a large drop.
Limitations
- Relies on machine-translated target-language captions, so translation noise is not directly addressed.
- Evaluated only on five target languages for images and eight for video, all with MT-based supervision.
Experimental setup
Datasets: Multi30K MSCOCO MSRVTT CC3M CC300K
Benchmarks: Multi30K MSCOCO MSRVTT
Metrics: mean Average Recall (mAR) text-to-video Recall@1
Baselines: UC2 MURAL MLA NRCCR DCOT CL2CM MMP C2KD
Builds on: MLA CLIP MAD-X UC2 CCLM
Closest papers by meaning
- CL2CM: Improving Cross-Lingual Cross-Modal Retrieval via Cross-Lingual Knowledge Transfer (AAAI 2024 · similarity 0.93)
- WikiDO: A New Benchmark Evaluating Cross-Modal Retrieval for Vision-Language Models (NeurIPS 2024 · similarity 0.91)
- Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion (ICLR 2025 · similarity 0.89)
- Beyond Global Similarity: Multi-Conditional Retrieval for Fine-Grained Cross-Modal Understanding (CVPR 2026 · similarity 0.89)
- UniAdapter: Unified Parameter-Efficient Transfer Learning for Cross-modal Modeling (ICLR 2024 · similarity 0.89)
- Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.