Research map / Vision-language and multimodal / Cross-modal learning
RankAlign: Unsupervised Vision-Language Representation Alignment via Rank Transformation
Enzhe Zhao, Marco Fiorucci, Lamberto Ballan
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RankAlign, an unsupervised method that aligns unpaired vision and language representations by comparing rank-transformed intra-modality similarity structures.
Problem
Unsupervised alignment of unpaired vision and language model representations; existing methods compare raw intra-modality similarity kernels, which are incommensurate across modalities due to differing scales and densities, leading to ill-conditioned optimization.
Approach
Transform each modality's similarity kernel into ranks before comparison, so that alignment operates on commensurate rank structures rather than raw similarity values, motivated by the Platonic Representation Hypothesis.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace raw kernels with rank-based ordinal kernels | BlindMatch with FAQ solver · CIFAR-100 N=100 | matching accuracy: 0.04 -> 0.66 | Improved |
| replace raw kernels with rank-based kernels | BlindMatch with OT solver · CIFAR-100 N=100 | matching accuracy: 0.01 -> 0.69 | Improved |
| replace raw kernels with rank-based kernels | BlindMatch with FHG solver · CIFAR-100 N=100 | matching accuracy: 0.15 -> 0.79 | Improved |
| rank transform | RankAlign · CIFAR-100 | runtime: FAQ N=100: 1.44 -> 2.70 s | Hurt |
| rank transform instead of z-score (ablation) | RankAlign · CIFAR-100 N=100 | matching accuracy: 0.00 -> 0.79 | Improved |
| rank transform instead of row-wise ranking (ablation) | RankAlign · CIFAR-100 N=100 | matching accuracy: 0.06 -> 0.79 | Improved |
| rank transform instead of quantile normalization (ablation) | RankAlign · CIFAR-100 N=50 | matching accuracy: 0.07 -> 0.98 | Improved |
| rank-based alignment instead of BlindMatch | RankAlign · LAION pseudo-class | matching accuracy: DINOv2 0.59 -> 0.96; CLIP 0.77 -> 0.68 | Mixed |
| rank-based alignment instead of BlindMatch | RankAlign · CheXpert subset | matching accuracy: 0.08 -> 0.65 (DeiT) | Improved |
| increase visual label corruption from 0 to 0.40 | RankAlign · ImageNet-100 N=100 | matching accuracy: 74.4 -> 41.5 | Hurt |
| increase visual label corruption from 0 to 0.40 | RankAlign · CIFAR-100 N=100 | matching accuracy: 78.7 -> 68.9 | Hurt |
| compare RankAlign with Procrustes and vanilla GW-OT | RankAlign · CIFAR-100 N=100 (CLIP) | matching accuracy: 0.03 -> 0.79 (vs GW-OT) | Improved |
Experimental setup
Baselines: direct similarity kernel alignment methods
Builds on: Platonic Representation Hypothesis
Closest papers by meaning
- Mind the Heads: Topological Representation Alignment for Multimodal LLMs (NeurIPS 2026 · similarity 0.88)
- DH-Set: Improving Vision-Language Alignment with Diverse and Hybrid Set-Embeddings Learning (CVPR 2025 · similarity 0.88)
- DeepAlign: Mitigating Modality Conflict through Modality-Specific Alignment (CVPR 2026 · similarity 0.88)
- Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment (NeurIPS 2024 · similarity 0.88)
- Topology-Aware Representation Alignment for Semi-Supervised Vision-Language Learning (NeurIPS 2026 · similarity 0.88)
- Do Vision and Language Encoders Represent the World Similarly? (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
