Research map / Large language models / AI and society
ReSi: A Comprehensive Benchmark for Representational Similarity Measures
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ReSi, a benchmark of six tests (prediction- and design-grounded) to evaluate 24 representational similarity measures across 14 graph, language, and vision architectures and seven datasets.
Problem
There is no comprehensive, well-grounded way to evaluate representational similarity measures; prior comparisons were ad hoc and covered few measures and criteria.
Approach
Grounds representational similarity in six tests: two grounded by predictions (correlation with accuracy difference and output difference) and four grounded by design (label randomization, shortcut affinity, augmentation, layer monotonicity), applied across graph, language, and vision models.
Key results
- No single similarity measure performs consistently best across domains; CKA ranks first in language but eighth in vision and 11th in graphs.
- Jaccard similarity emerged as the best-performing measure for the vision domain, and 2nd-order cosine and rank similarity perform well on graphs.
- Recommendations: distance correlation or CKA for language, Jaccard for vision, and neighborhood-based measures for graphs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use domain-specific rather than one universally best similarity measure | Representational similarity measure selection · Graph, language, and vision domains | Aggregated rank across tests | Mixed |
| Use neighborhood-based measures | Graph representations · Graph domain | Aggregated rank | Helped |
| Use CKA | Language representations · Language domain | Aggregated rank | Helped |
| Use Jaccard similarity | Vision representations · Vision domain | Aggregated rank | Helped |
| Use neighborhood-based measures to correlate similarity with predictions | BERT representations · SST2; Correlation to JSD Difference | Spearman correlation: 0.56 | Helped |
| Use RSMDiff to distinguish label-randomization groups | GraphSAGE representations · Flickr; Label Randomization | AUPRC: 0.92 | Helped |
| Use AlignCos to distinguish shortcut-affinity groups | GraphSAGE representations · Flickr; Shortcut Affinity | AUPRC: 1.00 | Helped |
| Use DistCorr to distinguish augmentation groups | BERT representations · SST2; Augmentation | AUPRC: 0.62 | Helped |
| Use PWCCA to measure layer monotonicity | BERT representations · SST2; Layer Monotonicity | Spearman correlation: 1.00 | Helped |
| Use OrthProc instead of ProcDist preprocessing | Representational similarity measures · Benchmark tests | Similarity measure performance | Mixed |
Surprising findings
- Rarely used measures such as Jaccard similarity outperform popular measures like CKA in the vision domain.
- Preprocessing alone (OrthProc vs ProcDist) substantially changes what properties similarity measures capture.
Limitations
- Grounding by design cannot fully control what models learn; behaviorally identical models may differ internally.
- Tests only focus on last-layer representations and single-modality settings.
- Not all existing similarity measures are included and preprocessing/hyperparameter effects were not explored.
Experimental setup
Datasets: Cora Flickr OGBN-Arxiv SST2 MNLI ImageNet100 CIFAR-100 ImageNet21k
Benchmarks: ReSi (six tests across graph, language, vision)
Metrics: Spearman correlation AUPRC conformity rate
Baselines: CKA SVCCA PWCCA RSA Orthogonal Procrustes Jaccard similarity
Builds on: Centered Kernel Alignment Representational Similarity Analysis Singular Vector CCA Projection-Weighted CCA
Closest papers by meaning
- Representational Similarity via Interpretable Visual Concepts (ICLR 2025 · similarity 0.87)
- Tracing Representation Progression: Analyzing and Enhancing Layer-Wise Similarity (ICLR 2025 · similarity 0.87)
- Objective drives the consistency of representational similarity across datasets (ICML 2025 · similarity 0.87)
- How Not to Stitch Representations to Measure Similarity: Task Loss Matching Versus Direct Matching (AAAI 2025 · similarity 0.86)
- Going Beyond Neural Network Feature Similarity: The Network Feature Complexity and Its Interpretation Using Category Theory (ICLR 2024 · similarity 0.86)
- Decoupling Semantic Similarity from Spatial Alignment for Neural Networks. (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.