Research map / Vision-language and multimodal / Human and embodied modeling
Decoupling Semantic Similarity from Spatial Alignment for Neural Networks.
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes semantic RSMs that make representational similarity matrices invariant to spatial permutation of concept vectors via optimal bipartite matching (with fast Batch-Optimal approximations), and shows better retrieval and better agreement with classifier output similarity than spatio-semantic RSMs.
Problem
Standard representational similarity matrices (RSMs) used with CKA couple semantic similarity with spatial alignment, so two images with the same semantic content at different spatial locations are judged dissimilar.
Approach
Compute sample-pair similarity after finding the optimal permutation (Hungarian/linear assignment) of spatial semantic concept vectors between two representations, making the RSM invariant to spatial permutations; approximate algorithms (Greedy, TopK-Greedy, Batch-Optimal) reduce runtime.
Key results
- On EgoObjects retrieval (5k database), permutation-invariant cosine similarity raises SAM ViT/B32 F1@1 from 0.614 to 0.804.
- Permutation-invariant cosine similarity gives more negative Pearson correlation with JSD than spatio-semantic similarity for 6 of 7 classifiers (e.g., ResNet18 -0.276 to -0.326).
- Batch-Optimal approximation (b=512) improves runtime by ~36x over the fastest optimal algorithm at spatial extent 4096 with under 8% error.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add permutation invariance to cosine-similarity retrieval | General feature extractors · EgoObjects test, database size 5.000 | F1@1: 0.280 -> 0.835; 0.827 -> 0.876; 0.750 -> 0.846; 0.784 -> 0.858; 0.614 -> 0.804 | Helped |
| Add permutation invariance to cosine-similarity retrieval | General feature extractors · Cityscapes validation-to-training retrieval | IoU: 0.662 -> 0.694; 0.689 -> 0.700; 0.679 -> 0.687; 0.691 -> 0.701; 0.690 -> 0.702 | Helped |
| Add permutation invariance to representational cosine similarity | ImageNet1k classifiers · ImageNet1k test, representational similarity vs predicted-probability JSD | Pearson correlation ρ: -0.276 -> -0.326; -0.248 -> -0.291; -0.192 -> -0.276; -0.134 -> -0.098; -0.046 -> -0.100; -0.138 -> -0.188; -0.012 -> -0 | Mixed |
| Add permutation invariance to inner-product retrieval | General feature extractors · Cityscapes validation-to-training retrieval | IoU: 0.664 -> 0.654; 0.686 -> 0.696; 0.677 -> 0.677; 0.693 -> 0.701; 0.687 -> 0.677 | Mixed |
| Use Batch-Optimal matching instead of optimal matching | Semantic RSM matching · ResNet101 on Tiny-ImageNet, spatial extent 4096 | relative similarity k/k_opt: 57.9 -> 92.8 | Helped |
| Use Batch-Optimal matching with batch size 128 instead of no matching | Semantic RSM matching · ResNet101 on Tiny-ImageNet, spatial extent 4096 | relative similarity k/k_opt: 57.9 -> 92.8 | Helped |
| Use Greedy matching rather than same-position assignment | Semantic RSM matching · ResNet18 on Tiny-ImageNet | matching quality | No clear effect |
| Match only the highest-norm concept vectors with TopK-Greedy | Semantic RSM matching · ResNet18 on Tiny-ImageNet | matching quality | Hurt |
| Use Batch-Optimal matching with batch size 128 | Semantic RSM matching · ResNet101 on Tiny-ImageNet | matching error: less than 10% | Helped |
Surprising findings
- Spatio-semantic RSMs on a ViT-B/16 barely recognize partially overlapping crops, and the RBF kernel gives positive correlation with output similarity for several ResNets.
Limitations
- Optimal permutation matching is NP-hard style per pair and O(S^3) per pair, making early high-resolution CNN layers and segmentation outputs costly.
- Correlations with classifier output similarity are generally low, and no quantitative benchmark exists for CKA-based similarity measures.
- Retrieval baselines are limited to methods applied directly to representations, not compared with established global-vector retrieval methods.
Experimental setup
Datasets: EgoObjects Tiny-ImageNet ImageNet-1k CIFAR100 Cityscapes
Benchmarks: EgoObjects retrieval Cityscapes retrieval ImageNet-1k classifier output similarity
Metrics: F1@1 IoU Pearson correlation with JSD Spearman correlation CKA runtime
Baselines: spatio-semantic RSM translation-invariant RSM no matching (same-position assignment)
Builds on: Centered Kernel Alignment Representational Similarity Matrices translation-invariant RSM (circular shift)
Closest papers by meaning
- Representational Similarity via Interpretable Visual Concepts (ICLR 2025 · similarity 0.88)
- Conditional Similarity Networks (CVPR 2017 · similarity 0.87)
- Objective drives the consistency of representational similarity across datasets (ICML 2025 · similarity 0.86)
- ReSi: A Comprehensive Benchmark for Representational Similarity Measures (ICLR 2025 · similarity 0.86)
- Tracing Representation Progression: Analyzing and Enhancing Layer-Wise Similarity (ICLR 2025 · similarity 0.85)
- Geometry-Aware Similarity Metrics for Neural Representations on Riemannian and Statistical Manifolds (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.