Research map / Label-efficient and robust learning / Recognition and long-tail
Discriminating image representations with principal distortions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes principal distortions, a variance-maximizing extension of generalized eigen-distortions based on Fisher information metrics, to find two image distortions that best differentiate N image representations, applied to early visual models and DNNs.
Problem
Comparing image representations (biological or artificial) by their local geometry, since models with similar global structure can differ strongly in local sensitivity to small image distortions, and existing generalized eigen-distortion methods only handle pairs of models.
Approach
Defines a pseudometric on local geometry from Fisher information matrices (log sensitivity ratios of two distortions) and chooses two distortions that maximize the variance of log sensitivity ratios across N models, analogous to PCA; this extends generalized eigen-distortions beyond pairwise comparisons.
Key results
- Principal distortions give diverse log sensitivity ratios across four nested early visual models, whereas random distortions do not; LGN and LN appear closest to human thresholds by visual inspection.
- AlexNet and ResNet50 are separated by architecture: AlexNet is more sensitive to distortion epsilon_1 (high-variability regions) and ResNet50 to epsilon_2 (smooth regions), consistent across 100 ImageNet base images.
- Adversarially trained models are separated from standard models by training type, whereas SIN-trained models separate by architecture rather than training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use principal distortions to compare AlexNet and ResNet50 representations | AlexNet and ResNet50 · ImageNet images | local sensitivity to principal distortions | Mixed |
| Train on Stylized ImageNet instead of standard ImageNet | AlexNet and ResNet50 · ImageNet images | local sensitivity geometry | No clear effect |
| Use adversarial training instead of standard training | AlexNet and ResNet50 · ImageNet images | local sensitivity geometry | Mixed |
| Compare EfficientNet-B0 and ViT using principal distortions | EfficientNet-B0 and ViT · ImageNet validation images | local sensitivity to principal distortions | Mixed |
| Add the OFF channel and gain-control operations (ablation) | LN, LG, LGG, and LGN early visual models · Kodak TID 2008 images | qualitative agreement with human distortion visibility | Mixed |
| Remove the image-gamut clipping constraint during distortion optimization (ablation) | Principal distortion optimization · ResNet50 and AlexNet comparison | principal distortion results | No clear effect |
Surprising findings
- Stylized ImageNet training does not change the local geometry of AlexNet or ResNet50 in a way that separates them from standard counterparts; architecture dominates.
- Adversarial training dramatically changes local geometry, separating models by training type rather than architecture.
Limitations
- Local linear analysis only holds in an infinitesimal neighborhood of the base image.
- Assumes additive Gaussian response noise for deterministic models, which may not represent neural responses.
- Qualitative comparisons only; no human psychophysics validation is presented.
Experimental setup
Datasets: TID-2008 ImageNet Stylized ImageNet ImageNet-9
Benchmarks: TID-2008
Metrics: log sensitivity ratio variance of log sensitivity ratios proportion of changed classifications
Baselines: generalized eigen-distortions eigen-distortions
Builds on: generalized eigen-distortions eigen-distortions Fisher-Rao metric
Closest papers by meaning
- Estimating Shape Distances on Neural Representations with Limited Samples (ICLR 2024 · similarity 0.87)
- The Unreasonable Effectiveness of Linear Prediction as a Perceptual Metric (ICLR 2024 · similarity 0.87)
- Objective drives the consistency of representational similarity across datasets (ICML 2025 · similarity 0.86)
- Connecting Neural Models Latent Geometries with Relative Geodesic Representations (NeurIPS 2025 · similarity 0.86)
- Visual Anagrams Reveal Hidden Differences in Holistic Shape Processing Across Vision Models (NeurIPS 2025 · similarity 0.86)
- Characterizing Universal Object Representations Across Vision Models (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.