Research map / Label-efficient and robust learning / Recognition and long-tail
When Does Closeness in Distribution Imply Representational Similarity? An Identifiability Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that small KL divergence between models in an identifiable softmax-embedding family does not guarantee representational similarity, and proposes a log-likelihood-variance distributional distance that bounds a PLS-SVD-based representational dissimilarity.
Problem
When do models with close output distributions learn similar internal representations? Classical linear identifiability assumes exact likelihood equality, and it is unclear whether approximate equality (small KL, near-maximal likelihood) implies representational similarity.
Approach
Proves that KL divergence can vanish while embeddings stay linearly dissimilar (via a norm-growing unembedding construction), then defines a log-likelihood-variance distance whose smallness provably bounds a PLS-SVD-based representational dissimilarity under a diversity condition.
Key results
- Constructed model pairs with KL divergence falling from 0.8866 to 0.0001 as unembedding norm rho goes from 3 to 18, while the LLV distance stays ~1.317 and m_CCA stays ~0.001.
- On CIFAR-10, two retrained models with test losses 1.19 and 1.18 and KL 1.13/1.12 have LLV distance ~1.55 and m_CCA 0.55.
- In synthetic experiments, wider networks (widths 16 to 256) yield lower mean and standard deviation of the LLV distance and higher representational similarity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase unembedding norm ρ | Fixed-representation model pair · Synthetic construction | KL divergence: 0.8866 -> 0.0001 | Helped |
| Increase unembedding norm ρ | Fixed-representation model pair · Synthetic construction | max d_SVD: 0.9996 -> 0.9998 | No clear effect |
| Increase network width | Synthetic 2D classification models · Synthetic 2D classification (4, 6, 10, and 18 labels) | LLV distance and representational dissimilarity | Helped |
| Train separate retrainings with similarly good performance | CIFAR-10 ResNet18 classifiers · CIFAR-10 | representational similarity (mean CCA): 0.55 | Hurt |
Surprising findings
- KL divergence between models can approach zero while their embeddings remain linearly dissimilar, so near-maximum-likelihood models can learn dissimilar representations.
- Identifiability results may not be robust to approximate likelihood maximization under KL divergence.
Limitations
- The distance bound is linear in epsilon and not tight, and requires the label set to contain all classes but one.
- The diversity condition may fail for image classifiers with few classes relative to representation dimension.
- Analysis covers only embedding and unembedding layers, not intermediate representations; no error bars reported.
Experimental setup
Datasets: CIFAR-10 synthetic 2D Gaussian classification data
Benchmarks: CIFAR-10 synthetic 2D classification
Metrics: KL divergence log-likelihood variance distance mean canonical correlation PLS-SVD dissimilarity test loss accuracy
Baselines: mean canonical correlation (m_CCA)
Builds on: linear identifiability (Roeder et al. 2021) PLS-SVD canonical correlation analysis (CCA)
Closest papers by meaning
- Tracing Representation Progression: Analyzing and Enhancing Layer-Wise Similarity (ICLR 2025 · similarity 0.88)
- Objective drives the consistency of representational similarity across datasets (ICML 2025 · similarity 0.87)
- Representational Similarity via Interpretable Visual Concepts (ICLR 2025 · similarity 0.87)
- Not all solutions are created equal: An analytical dissociation of functional and representational similarity in deep linear neural networks (ICML 2025 · similarity 0.85)
- ReSi: A Comprehensive Benchmark for Representational Similarity Measures (ICLR 2025 · similarity 0.85)
- Estimating Shape Distances on Neural Representations with Limited Samples (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.