Research map / Vision-language and multimodal / Cross-modal learning
Learning Shared Representations from Unpaired Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns a shared text-image embedding almost exclusively from unpaired data by combining spectral embeddings of unimodal random walks, CCA on very few pairs, and an MMD residual network (SUE).
Problem
Learning shared multimodal representations typically requires large amounts of paired data, which is costly or impossible to obtain in many domains; the paper asks whether shared spaces can be learned from almost exclusively unpaired data plus very few pairs.
Approach
Train independent SpectralNet spectral embeddings on each unimodal representation (random-walk eigenfunctions, argued to be modality-invariant), align them linearly with CCA on a handful of pairs, and refine non-linearly with an MMD-trained residual network that needs no pairs.
Key results
- On MSCOCO with 100 pairs, SUE R@10 text-to-image 33.25 vs 12.20 for contrastive, about +257% average relative improvement over contrastive learning with the same pairs.
- Zero-shot ImageNet classification (50-sample test set) with SUE trained on Flickr30k with 500 pairs reaches 88.31% accuracy versus 99.26% for CLIP trained on ~400M pairs.
- Ablation on Flickr30k: T2I R@10 rises from 4.75 (CCA+MMD without SE) to 32.75 with SE, showing spectral embedding is essential.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SUE with unpaired data and few paired examples instead of contrastive learning with the same pair count | MSCOCO retrieval · MSCOCO | R@10: 13.00 -> 34.25; 12.20 -> 33.25 | Helped |
| Use SUE with unpaired data and few paired examples instead of contrastive learning with the same pair count | Flickr30k retrieval · Flickr30k | R@10: 16.20 -> 32.00; 15.00 -> 32.75 | Helped |
| Use SUE with unpaired data and few paired examples instead of contrastive learning with the same pair count | Polyvore retrieval · Polyvore | R@10: 22.5 -> 32.25; 23.00 -> 32.00 | Helped |
| Use SUE with unpaired data and few paired examples instead of contrastive learning with the same pair count | Edges2Shoes retrieval · Edges2Shoes | R@10: 14.00 -> 25.25; 12.80 -> 27.00 | Helped |
| Use SUE with unpaired data and few paired examples instead of contrastive learning with the same pair count | Handwritten retrieval · Handwritten | R@10: 28.00 -> 79.00; 30.50 -> 78.00 | Helped |
| Add spectral embedding (SE) before CCA and MMD (ablation) | SUE · Flickr30k T2I | R@10: 4.75 -> 32.75 | Helped |
| Add spectral embedding (SE) before CCA and MMD (ablation) | SUE · MSCOCO T2I | R@10: 9.75 -> 33.25 | Helped |
| Add CCA to SE + MMD (ablation) | SUE · Flickr30k T2I | R@10: 5.50 -> 32.75 | Helped |
| Replace SE with an Auto-Encoder (ablation) | SUE · Flickr30k retrieval | R@10: 33.50 -> 2.75 | Hurt |
| Increase the amount of unpaired data | SUE · Flickr30k image retrieval | R@10 | Helped |
| Increase paired samples beyond the minimum needed for CCA | SUE · Flickr30k image retrieval | R@10 | No clear effect |
Surprising findings
- Adding unpaired samples substantially improves retrieval, and SUE with 100 pairs outperforms contrastive learning requiring an order of magnitude more pairs.
- Replacing SpectralNet with an autoencoder in the pipeline yields near-random retrieval (R@10 2.75 vs random ~2.5).
Limitations
- Does not match state-of-the-art models trained on massive paired datasets.
- Evaluated only on a small set of modalities and tasks; generalization to video, speech, or scientific data untested.
- Retrieval performance remains low in absolute terms (R@1 around 4-6%).
Experimental setup
Datasets: Flickr30k MSCOCO Polyvore Edges2Shoes Handwritten caption-FFHQ ImageNet Office31 FFHQ
Benchmarks: MSCOCO Flickr30k Polyvore Edges2Shoes Handwritten ImageNet Office31
Metrics: Recall@1 Recall@5 Recall@10 mAP zero-shot accuracy classification accuracy Relative CLIP Score
Baselines: Contrastive (CLIP-style MLP on same pairs) CSA MMD SAIL CLIP
Builds on: SpectralNet CCA MMD CLIP CSA
Closest papers by meaning
- Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR 2024 · similarity 0.88)
- Aligning Multimodal Representations through an Information Bottleneck (ICML 2025 · similarity 0.87)
- An Information Criterion for Controlled Disentanglement of Multimodal Data (ICLR 2025 · similarity 0.87)
- RePercENT: Scaling Disentangled Representation Learning Beyond Two Modalities (NeurIPS 2026 · similarity 0.86)
- Identifiable Shared Component Analysis of Unpaired Multimodal Mixtures (NeurIPS 2024 · similarity 0.86)
- Noisy Correspondence Rectification via Asymmetric Similarity Learning (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.