Research map / Vision-language and multimodal / Cross-modal learning

CSA: Data-efficient Mapping of Unimodal Features to Multimodal Features

ICLR 2025 · Vision-language and multimodal › Cross-modal learning · approach: Unimodal-to-multimodal representation transfer · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes canonical similarity analysis (CSA), which maps features from two pre-trained unimodal encoders into a shared multimodal space via CCA plus a weighted cosine similarity, matching CLIP-like zero-shot performance with ~35k paired examples and no neural network training.

Problem

Multimodal encoders such as CLIP require huge amounts of paired multimodal data and GPU training, and are sensitive to noisy web data; the goal is to bridge two pre-trained unimodal encoders into a shared multimodal space data-efficiently.

Approach

Use CCA on features from two frozen unimodal encoders to find correlated bases, then compute a weighted cosine similarity over the top-s canonical dimensions (weights = correlation coefficients), so that the similarity mimics CLIP's cosine similarity without training any neural network.

Key results

• On ImageNet classification CSA matches CLIP with about 35,000 training samples, while ASIF needs millions of data points.
• On Flickr30k text-to-image retrieval, CSA reaches 44.7% Precision@1 versus CLIP's 79.5% and ASIF's 0.1%.
• On COSMOS misinformation detection, CSA outperforms CLIP-based baselines in AUC and is only beaten by the supervised COSMOS method; CSA keeps 70% accuracy with 50% shuffled ImageNet training labels.

Measured effects reported in the paper

Surprising findings

• CSA fails to match CLIP on cross-modal retrieval on Flickr30k despite matching it on classification and detection tasks.
• LLaVA-13B completely fails the COSMOS misinformation task with a 100% false positive rate.

Limitations

• Limited to two modalities, unlike ImageBind's six.
• Performance degrades when unimodal encoders are not foundation models (e.g., trained on MNIST).
• Underperforms CLIP on cross-modal retrieval over large reference sets.

Experimental setup

Datasets: ImageNet Leafy Spurge Flickr30k COSMOS MusicCaps KITTI handwritten alphabet time series

Benchmarks: ImageNet Leafy Spurge Flickr30k COSMOS MusicCaps KITTI handwritten alphabet time series

Metrics: accuracy mAP@5 precision@1 precision@5 AUC ROC

Baselines: CLIP ASIF LLaVA COSMOS supervised method CLAP Lip-loc

Builds on: CLIP ASIF canonical correlation analysis contrastive loss

Closest papers by meaning

• Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignment (CVPR 2025 · similarity 0.89)
• With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide You (NeurIPS 2025 · similarity 0.88)
• UNIALIGN: Scaling Multimodal Alignment within One Unified Model (CVPR 2025 · similarity 0.86)
• MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer (ICLR 2026 · similarity 0.85)
• MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding (ICLR 2025 · similarity 0.85)
• UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them All (CVPR 2024 · similarity 0.85)