Research map / Vision-language and multimodal / Cross-modal learning
SOTAlign: Semi-Supervised Alignment of Unimodal Vision and Language Models via Optimal Transport
Simon Roschmann, Paul Krzakala, Sonia Mazelet, Quentin Bouniot, Zeynep Akata
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace CKA divergence with KLOT in the regularizer (ablation) | SOTAlign with CCA teacher · COCO MeanR@1 | MeanR@1: 23.5 -> 30.3 | Improved |
| replace InfoNCE divergence with KLOT in the regularizer (ablation) | SOTAlign with CCA teacher · COCO MeanR@1 | MeanR@1: 24.1 -> 30.3 | Improved |
| add unpaired-data KLOT regularization to a CCA linear teacher (ablation) | CCA alignment · COCO MeanR@1 | MeanR@1: 21.5 -> 30.3 | Improved |
| use CCA teacher instead of contrastive linear teacher with KLOT (ablation) | SOTAlign · COCO MeanR@1 | MeanR@1: 28.5 -> 30.3 | Improved |
| use CKA with a contrastive teacher (ablation) | SOTAlign · COCO MeanR@1 | MeanR@1: 24.2 -> 24.2 | No clear effect |
| add unpaired images and text via SOTAlign vs supervised SAIL | 10k-pair alignment · ImageNet-1k zero-shot | top-1 accuracy: 35.6 -> 46.1 | Improved |
| add unpaired samples as extra negatives or pseudo-labels (semi-supervised SAIL, NNCLR, S-CLIP) | supervised baselines · COCO retrieval | T2I Recall@1: 21.0 -> 20.7 (SAIL) | No clear effect |
| increase number of paired samples (ablation) | SOTAlign vs SAIL · COCO MeanR@1 | MeanR@1: gains up to +10% between 10^3 and 10^4 pairs | Mixed |
| use out-of-distribution unpaired data (CC12M, COCO, ImageNet-1k, synthetic captions) (ablation) | SOTAlign · cross-dataset unpaired data | retrieval and classification: Pearson r = -0.76 with shift | Improved |
| replace unpaired data with Gaussian-sampled embeddings (ablation) | SOTAlign · COCO retrieval | retrieval | No clear effect |
| use CC3M pairs with synthetic captions (ablation) | SOTAlign · COCO T2I R@1 | Recall@1: 26.5 -> 31.3 | Improved |
| use DINOv3 instead of DINOv2 vision encoder with NV-Embed-v2 (ablation) | SOTAlign · ImageNet-1k zero-shot | top-1 accuracy: 42.5 -> 46.1 | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- S-CLIP — S-CLIP: Semi-supervised Vision-Language Learning using Few Specialist Captions (NeurIPS 2023) · better on 1 of 1 shared result
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Topology-Aware Representation Alignment for Semi-Supervised Vision-Language Learning (NeurIPS 2026 · similarity 0.89)
- Assessing and Learning Alignment of Unimodal Vision and Language Models (CVPR 2025 · similarity 0.89)
- Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignment (CVPR 2025 · similarity 0.88)
- SineProject: Machine Unlearning for Stable Vision-Language Alignment (CVPR 2026 · similarity 0.87)
- DH-Set: Improving Vision-Language Alignment with Diverse and Hybrid Set-Embeddings Learning (CVPR 2025 · similarity 0.87)
- DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
