Research map / Label-efficient and robust learning / Self-supervised and contrastive
Objective drives the consistency of representational similarity across datasets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measures how pairwise CKA similarities between 64 vision models transfer across 23 datasets, finding training objective (especially SSL) drives consistency and that similarity-performance links depend on dataset type.
Problem
Representational similarity between models is usually measured on a single dataset, and it is unclear whether pairwise similarities transfer across datasets, what factors drive that consistency, and how it relates to downstream performance differences.
Approach
Quantify similarity consistency as the Pearson correlation between vectors of pairwise model similarities (CKA) computed on two datasets, aggregated over model sets grouped by training factors and over all dataset pairs, and relate these to performance gaps.
Key results
- Mean Pearson correlation of pairwise model similarities across dataset pairs is 0.756 (std 0.124).
- Self-supervised model sets show the highest similarity consistency across datasets, while supervised model sets show the weakest, especially against image-text models.
- Similarity and performance gap are strongly negatively correlated on single-domain and structured datasets, but only weakly on multi-domain datasets such as ImageNet-1k.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use self-supervised vision models rather than supervised or image-text models | Pairwise representational similarity comparisons across vision models · Across datasets | Similarity consistency | Helped |
| Use supervised models rather than self-supervised models | Pairwise representational similarity comparisons across vision models · Across datasets | Similarity consistency | Hurt |
| Use image-text models rather than self-supervised models | Pairwise representational similarity comparisons across vision models · Across datasets | Similarity consistency | Hurt |
| Compare models within the same size category rather than across different size categories | Pairwise representational similarity comparisons across vision models · Across datasets | Similarity consistency | Helped |
| Change model architecture between convolutional and transformer networks | Pairwise representational similarity comparisons across vision models · Across datasets | Similarity consistency | No clear effect |
| Evaluate similarity consistency within natural multi-domain datasets rather than across dataset categories | All model pairs · Natural multi-domain datasets | Pearson correlation of representational similarities | Helped |
| Evaluate image-text and supervised model pairs across multi-domain rather than single-domain datasets | Image-text and supervised model pairs · Natural multi-domain vs. natural single-domain datasets | Similarity consistency | Helped |
| Evaluate self-supervised model pairs across different dataset types | Self-supervised model pairs · Across dataset types | Similarity consistency | No clear effect |
| Use CKA RBF (σ=0.2) rather than CKA linear | Representational similarity consistency analysis · Across model sets and dataset pairs | Similarity consistency patterns | No clear effect |
| Use CKA RBF (σ=0.2) rather than CKA linear | Image-text vs. supervised model pairs · Across datasets | Similarity consistency | Mixed |
| Use CKA linear rather than CKA RBF (σ=0.2) | Representational similarity estimates · ImageNet-1k subsets | Sample stability: 10{,}000 vs. 30{,}000 | Helped |
| Use CKA RBF (σ=0.2) rather than CKA linear | Representational similarity estimates · ImageNet-1k subsets | Sample stability: 10{,}000 vs. 30{,}000 | Hurt |
Surprising findings
- Model architecture and model size have minor effects on similarity consistency, while training data size matters less than the objective.
- Image-text and supervised anchor models correlate more strongly with SSL models than with their own training-objective category on global similarity.
Limitations
- Limited number of structured datasets restricts conclusions about the structured category.
- Training objective is confounded with training data in the model set, so effects may not be fully separated.
- Layer choice differs across models (e.g., average pooling vs. norm vs. penultimate), which may affect comparisons.
Experimental setup
Datasets: ImageNet-1k CIFAR-10 CIFAR-100 Caltech101 Country211 STL-10 Pascal VOC 2007 Breeds (Entity-30, Living-17, NonLiving-26) Cars FGVC Aircraft Flowers GTSRB
Benchmarks: CLIP benchmark datasets VTAB datasets Breeds
Metrics: Pearson correlation of pairwise similarity vectors CKA linear CKA RBF RSA Spearman top-1 accuracy performance gap
Baselines: RSA Spearman CKA RBF
Builds on: Platonic Representation Hypothesis Centered Kernel Alignment Representational Similarity Analysis
Closest papers by meaning
- Characterizing Universal Object Representations Across Vision Models (NeurIPS 2026 · similarity 0.88)
- Functional Alignment Can Mislead: Examining Model Stitching (ICML 2025 · similarity 0.87)
- Bridging Critical Gaps in Convergent Learning: How Representational Alignment Evolves Across Layers, Training, and Distribution Shifts (NeurIPS 2025 · similarity 0.87)
- When Does Closeness in Distribution Imply Representational Similarity? An Identifiability Perspective (NeurIPS 2025 · similarity 0.87)
- Position: The Platonic Representation Hypothesis (ICML 2024 · similarity 0.87)
- ReSi: A Comprehensive Benchmark for Representational Similarity Measures (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.