Research map / Vision-language and multimodal / Human and embodied modeling
On the rankability of visual embeddings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that many frozen visual embeddings (7 encoders, 9 datasets, 7 attributes) are "rankable": a linear rank axis preserves ordinal attribute order, and extreme-pair or few-shot estimates can recover it cheaply.
Problem
Whether visual embedding spaces encode continuous ordinal attributes (age, crowd count, head pose, aesthetics, recency) along linear directions, so that global ranking by projection is possible, and how cheaply such rank axes can be recovered.
Approach
Defines rankability as the existence of a linear rank axis whose projection preserves attribute order, measures it with Spearman correlation against no-train, nonlinear and finetuned reference points, and shows rank axes can be estimated from two extreme examples (difference of cluster means) without continuous labels.
Key results
- Averaged over 7 encoders, linear rank axes reach Spearman 0.861 on Adience age vs 0.266 no-train lower bound, close to nonlinear (0.878) and finetuned (0.910).
- On Adience, a rank axis from just two extreme samples reaches Spearman ~0.75 on average, while few-shot labelled training with the same tiny sample size is near 0.
- Zero-shot text-difference prompts reach 0.782 on Adience vs 0.917 for the linear upper bound, showing language-based axes lag behind.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use pretrained visual embeddings and fit a linear rank axis rather than using a randomly initialized encoder | Visual encoders across 9 datasets · Adience (Age) | SRCC: 0.266 -> 0.861 | Helped |
| Use a nonlinear regressor instead of a linear rank axis | Visual embeddings · UTKFace (Age) | SRCC: 0.766 -> 0.776 | Helped |
| Use a linear rank axis instead of nonlinear regression | Visual embeddings · BIWI Kinect (Yaw) | SRCC: 0.434 -> 0.597 | Hurt |
| Use a linear rank axis instead of finetuning the encoder | Visual encoders · BIWI Kinect (Roll) | SRCC: 0.218 -> 0.859 | Hurt |
| Use CLIP rather than non-CLIP encoders | Visual embedding rankability · Adience (Age) | SRCC: 0.871 -> 0.928 | Helped |
| Use CLIP rather than a non-CLIP encoder | Visual embedding rankability · UCF-QNRF (Crowd count) | SRCC: 0.864 -> 0.870 | No clear effect |
| Use DINOv2 rather than the best CLIP encoder | Visual embedding rankability · BIWI Kinect (Yaw) | SRCC: 0.440 -> 0.804 | Helped |
| Use extreme samples without continuous labels rather than continuous-label few-shot training | Rank-axis estimation · Adience (Age), up to ~1k training samples | SRCC: close to 0 -> ~0.75 | Helped |
| Use text-difference prompts instead of one prompt | Zero-shot rank-axis search · Adience (Age) | SRCC: 0.670 -> 0.782 | Helped |
| Use text-difference prompts instead of one prompt | Zero-shot rank-axis search · UCF-QNRF (Crowd count) | SRCC: 0.523 -> 0.315 | Hurt |
| Transfer an age rank axis learned on Adience to UTKFace rather than training on UTKFace | Age rank axis · UTKFace (Age) | SRCC: 0.81 -> 0.68 | Hurt |
| Transfer an age rank axis learned on UTKFace to Adience rather than training on Adience | Age rank axis · Adience (Age) | SRCC: 0.91 -> 0.55 | Hurt |
Surprising findings
- Two extreme examples can outperform few-shot regression with continuous labels when the training set is at most about 1k samples.
- Yaw and roll head-pose angles are poorly rankable with linear axes for most encoders (e.g. roll 0.218 average), despite high finetuned upper bounds.
Limitations
- Empirical study over a limited set of seven attributes; no theoretical account of why rankability emerges.
- Finetuned upper bounds may be suboptimal due to standard hyperparameter choices.
- Zero-shot prompt search is limited to GPT-generated prompts and may miss better prompts.
Experimental setup
Datasets: UTKFace Adience UCF-QNRF ShanghaiTech-A ShanghaiTech-B BIWI Kinect AVA KonIQ-10k Historical Color Images
Benchmarks: UTKFace Adience UCF-QNRF ShanghaiTech-A ShanghaiTech-B BIWI Kinect AVA KonIQ-10k Historical Color Images
Metrics: Spearman rank correlation MAE cosine similarity
Baselines: No-train random encoder Nonlinear 2-layer MLP probe Finetuned encoder Yu et al. MiVOLO OrdinalCLIP CLIP-EBC CrowdCLIP
Builds on: CLIP DINOv2 steering vectors OrdinalCLIP
Closest papers by meaning
- Scaling Up Ordinal Embedding: A Landmark Approach (ICML 2019 · similarity 0.88)
- Geometric Order Learning for Rank Estimation (NeurIPS 2022 · similarity 0.86)
- Landmark Ordinal Embedding (NeurIPS 2019 · similarity 0.85)
- Finite Sample Prediction and Recovery Bounds for Ordinal Embedding (NeurIPS 2016 · similarity 0.85)
- Not Only Text: Exploring Compositionality of Visual Representations in Vision-Language Models (CVPR 2025 · similarity 0.84)
- The Unreasonable Effectiveness of Linear Prediction as a Perceptual Metric (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.