Research map / Learning theory and algorithms / Prediction and decision losses
Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Pure Square Euclidean Distance (PSED), a loss that subtracts the expected Euclidean distance under label permutations from SED to remove random consistency, and uses it as a representation-layer loss alongside cross-entropy in fully connected networks.
Problem
Quantifying and training for class-level discriminability of learned representations; existing similarity-matrix measures (dinfor, SED) are biased by random consistency and fail to capture class structure.
Approach
Apply the pure consistency measure framework to the Square Euclidean Distance between the similarity matrix and the true adjacency matrix: subtract its expectation over uniform label permutations, which has a closed-form O(kn^2) solution, yielding PSED as an unbiased, heterogeneous loss.
Key results
- On Yeast with 8 layers, CE-PSED achieves F-measure 0.4116 versus 0.1662 for CE, while on 3-layer Yeast it reaches 0.5903 versus 0.5475 for CE.
- On Pendigits (3-layer), CE-PSED accuracy is 0.9908 versus 0.7761 for CE and 0.9611 for CE-SED.
- CE-PSED is reported as best on most of 25 datasets, with a Friedman/Nemenyi test showing significantly lower average rank for CE-PSED.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add PSED loss to the representation layer instead of the strongest listed baseline loss | CE-PSED · Mpeg | accuracy: 0.6538±0.0004 -> 0.7338±0.0002 | Helped |
| Add PSED loss to the representation layer instead of the strongest listed baseline loss | CE-PSED · Mnist | accuracy: 0.9046±0.0000 -> 0.9062±0.0000 | Helped |
| Add PSED loss to the representation layer instead of the strongest listed baseline loss | CE-PSED · Pendigits | accuracy: 0.9611±0.0000 -> 0.9908±0.0000 | Helped |
| Add PSED loss to the representation layer instead of the strongest listed baseline loss | CE-PSED · Caltech-101 | accuracy: 0.3561±0.0001 -> 0.5005±0.0001 | Helped |
| Add PSED loss to the representation layer instead of the strongest listed baseline loss | CE-PSED · ImageNet | accuracy: 0.9753±0.0000 -> 0.9762±0.0000 | Helped |
| Add PSED loss to the representation layer instead of the strongest listed baseline loss | CE-PSED · Mpeg | F-measure: 0.6426±0.0004 -> 0.7244±0.0003 | Helped |
| Add PSED loss to the representation layer instead of the strongest listed baseline loss | CE-PSED · Mnist | F-measure: 0.9044±0.0000 -> 0.9061±0.0000 | Helped |
| Add PSED loss to the representation layer instead of the strongest listed baseline loss | CE-PSED · Pendigits | F-measure: 0.9610±0.0000 -> 0.9908±0.0000 | Helped |
| Add PSED loss to the representation layer instead of the strongest listed baseline loss | CE-PSED · Caltech-101 | F-measure: 0.2814±0.0001 -> 0.4708±0.0001 | Helped |
| Add PSED loss to the representation layer instead of the strongest listed baseline loss | CE-PSED · ImageNet | F-measure: 0.9754±0.0000 -> 0.9762±0.0000 | Helped |
| Increase network depth from 3 to 5 layers (ablation) | CE-PSED · Yeast | F-measure: 0.5903 -> 0.4951 | Hurt |
| Increase network depth from 3 to 8 layers (ablation) | CE-PSED · Yeast | F-measure: 0.5903 -> 0.4116 | Hurt |
Surprising findings
- Plain CE and CE-SED degrade sharply with depth on Yeast (F-measure ~0.15-0.17 at 5 and 8 layers), which the authors attribute to feature representation collapse.
- On ImageNet and Mnist the CE-PSED advantage is small or absent in some cases, e.g. CE-inform scores higher than CE-PSED on ImageNet accuracy (0.9753 vs 0.9762 is actually lower for inform), and CE scores 0.9711 vs 0.9607 on dataset 11.
Limitations
- Experiments use only fully connected networks on tabular and image-feature data; the ViT and CNN variants are only described in the appendix.
- Optimization convergence properties of PSED are not analyzed (stated as future work).
- Results rely on VGG and MoCo v3 extracted features for image datasets rather than end-to-end training.
Experimental setup
Datasets: Wine Quality Blood Transfusion Service Center Energy Efficiency Tic-Tac-Toe Endgame Oocytes-Merluccius-Nucleus-4d QSAR Biodegradation Yeast Semeion Handwritten Digit Steel Plates Faults Cardiotocography Ozone Level Detection SkillCraft1 Master Table
Benchmarks: Yeast Pendigits Caltech-101 ImageNet MNIST Ringnorm Mushroom Nursery Adult
Metrics: accuracy F-measure Euclidean distance between similarity matrix and YY^T information entropy Friedman test Nemenyi test
Baselines: CE CE-SED CE-inform (1-dinfor)
Builds on: Pure Consistency Measure framework Square Euclidean Distance similarity measure d_infor (Brockmeier et al. 2017) exponential Orlicz norm concentration inequality (Escande 2024)
Closest papers by meaning
- Volume-Aware Distance for Robust Similarity Learning (ICML 2025 · similarity 0.87)
- Trading off Consistency and Dimensionality of Convex Surrogates for Multiclass Classification (NeurIPS 2024 · similarity 0.86)
- Towards an Explainable Comparison and Alignment of Feature Embeddings (ICML 2025 · similarity 0.86)
- Euclidean Embedding of Data Using Local Distances (NeurIPS 2026 · similarity 0.85)
- Threshold-Consistent Margin Loss for Open-World Deep Metric Learning (ICLR 2024 · similarity 0.85)
- Estimating Shape Distances on Neural Representations with Limited Samples (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.