Research map / Vision-language and multimodal / Human and embodied modeling
Retrieval-based Disentangled Representation Learning with Natural Language Supervision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VDR, a sparse lexical bi-encoder that uses natural-language counterparts of data as supervision to learn vocabulary-space disentangled representations, evaluated on BEIR text retrieval and cross-modal retrieval (MS COCO, Flickr30k, ImageNet).
Problem
Disentangled representation learning lacks ground-truth factors of variation, and existing supervision methods are task-specific or require structured annotations. The paper also seeks to extend lexical retrievers to non-text (image) data.
Approach
Represent both data and their natural-language counterparts in a BERT-vocabulary sparse space (elu1p weights with top-k and bag-of-words gating), align them with a symmetric contrastive loss plus a nonparametric bag-of-words term, and add a contrastive mask to supervise inactive dimensions, so the vocabulary dimensions become interpretable factors.
Key results
- VDR_t2t reaches 44.5 average NDCG@10 on 12 BEIR datasets zero-shot, versus 35.8 for DPR and 42.4 for SPLADE.
- VDR_cm achieves 42.7 R-mean on MS COCO and 63.4 on Flickr30k, above CLIP-BERT (37.4 and 57.4) under identical training.
- In human evaluation, 92% of VDR token sets were judged satisfactory versus 85% for top-5 BLIP caption tokens.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VDR_t2t instead of DPR | text-to-text retrieval · BEIR zero-shot (12-dataset Avg.) | NDCG@10: 35.8 -> 44.5 | Helped |
| Use VDR_cm instead of CLIP-BERT | cross-modal retrieval · MS COCO | R-mean: 37.4 -> 42.7 | Helped |
| Use VDR_cm instead of CLIP-BERT | cross-modal retrieval · Flickr30k | R-mean: 57.4 -> 63.4 | Helped |
| Use VDR_cm instead of CLIP-BERT | cross-modal retrieval · ImageNet | Top1: 32.4 -> 38.7 | Helped |
| Use VDR_cm instead of DeCLIP | cross-modal retrieval · Flickr30k | R-mean: 65.3 -> 63.4 | Hurt |
| Use binary bag-of-words query inference | VDR_t2t · BEIR zero-shot (12-dataset Avg.) | NDCG@10: 44.5 -> 44.6 | No clear effect |
| Replace elu1p activation with ReLU (ablation) | VDR_t2t · BEIR zero-shot (12-dataset Avg.) | NDCG@10: 44.5 -> 43.0 | Hurt |
| Add the nonparametric BoW loss (ablation) | VDR_t2t · BEIR zero-shot (12-dataset Avg.) | NDCG@10: 43.5 -> 44.5 | Helped |
| Replace elu1p activation with ReLU (ablation) | VDR_t2t^α · BEIR zero-shot (12-dataset Avg.) | NDCG@10: 44.6 -> 43.8 | Hurt |
| Add the nonparametric BoW loss (ablation) | VDR_t2t^α · BEIR zero-shot (12-dataset Avg.) | NDCG@10: 40.3 -> 44.6 | Helped |
| Switch from BERT to multilingual BERT vocabulary (ablation) | VDR · BEIR zero-shot (12-dataset Avg.) | NDCG@10: 44.5 -> 42.6 | Hurt |
| Increase activation amount from k=512 to k=768 (ablation) | VDR cross-modal · ImageNet | Top1: 38.7 -> 38.7 | No clear effect |
Surprising findings
- Training the disentanglement head's projection from scratch on the passage side (no pretrained MLM projection) fails to converge, dropping NDCG@10 to 0.2 and falling below the untrained BERT lexical baseline.
- Nonparametric binary bag-of-words query retrieval (no neural forward pass) matches or beats most advanced dense baselines on BEIR, though it underperforms on cross-modal retrieval.
Limitations
- Cross-modal nonparametric inference is limited because images contain information not explicit in captions.
- Tokenizer (BERT WordPiece) splits multi-token concepts such as 'giraffe' into subword pieces, hindering interpretability.
- Advanced retrieval techniques (hard-negative mining, distillation, retrieval pretraining) were not integrated due to resource constraints.
Experimental setup
Datasets: MS MARCO YFCC15M BEIR ImageNet MS COCO Flickr30k
Benchmarks: BEIR ImageNet MS COCO Flickr30k
Metrics: NDCG@10 Recall@K R-mean Top-1/Top-5 accuracy latency per query human interpretation rate
Baselines: BM25 SPLADE DPR ANCE UnifieR Contriever SimLM MASTER RetroMAE LexMAE E5 CLIP
Builds on: DPR SPLADE CLIP BERT DeCLIP
Closest papers by meaning
- Graph-based Unsupervised Disentangled Representation Learning via Multimodal Large Language Models (NeurIPS 2024 · similarity 0.90)
- Disentangled Representation Learning via Flow Matching (NeurIPS 2026 · similarity 0.87)
- Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations (ICML 2019 · similarity 0.87)
- An Information Criterion for Controlled Disentanglement of Multimodal Data (ICLR 2025 · similarity 0.87)
- Enriching Disentanglement: From Logical Definitions to Quantitative Metrics (NeurIPS 2024 · similarity 0.85)
- Transferring disentangled representations: bridging the gap between synthetic and real images (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.