Research map / Label-efficient and robust learning / Recognition and long-tail
Balanced Data, Imbalanced Spectra: Unveiling Class Disparities with Spectral Imbalance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces spectral imbalance (per-class eigenvalue differences) as a source of class gaps on balanced data, with exact per-class error theory for a high-dimensional two-class GMM and a spectral quantile score plus test-time augmentation ensembling for 11 pretrained ImageNet encoders.
Problem
Class performance gaps persist even on sample-balanced datasets, including with state-of-the-art pretrained encoders, and their source is not well understood.
Approach
Attributes per-class error gaps to differences in per-class covariance spectra; derives exact asymptotic per-class error for ERM in a GMM with class-specific covariances, and uses a per-class spectral offset (power-law fit) and a Spectral Quantile Score to diagnose encoders and select augmentations at test time without retraining.
Key results
- Spectral Quantile Score correlates with empirical class bias across 11 encoders with PCC 0.90.
- Average maximum negative correlation of -0.69 between class accuracy and individual eigenvalues across 11 models; spectral offset a correlates at -0.68.
- Spectral re-factoring ensemble over 16 augmentations improves Zoom Blur (65.01%) and Snow (65.12%) to 67.616% combined, with average class gap improvement of 3.52%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| spectral re-factoring ensemble selecting per-class augmentation by spectral offset | Zoom Blur and Snow augmentations on pretrained encoder · ImageNet val | top-1 accuracy: +2.55% average over single augmentations; class gap improved 3.52% | Helped |
| spectral re-factoring ensemble over all augmentation pairs | augmentation pairs (15x16=240 runs) · ImageNet val | average and worst-augmentation performance: +2.6% average, +5.9% worse augmentation | Helped |
| increase eigenvalue scale of class 1 relative to class -1 (theory GMM) (ablation) | spectrally imbalanced Gaussian mixture model · GMM simulation | class gap | Hurt |
| per-class eigenvalue magnitude (larger spectrum) | pretrained encoders on ImageNet · ImageNet val | Pearson correlation with class accuracy: average max negative correlation -0.69 across 11 models | Hurt |
| larger power-law offset a of class spectrum | pretrained encoders on ImageNet · ImageNet val | Pearson correlation with class accuracy: average -0.68 across 11 models | Hurt |
| power-law decay rate b of class spectrum | pretrained encoders on ImageNet · ImageNet val | correlation with class accuracy | No clear effect |
Surprising findings
- Power-law decay rate b does not strongly correlate with class-wise accuracy, while offset a does.
- The first few eigenvalues do not show strong correlation with class accuracy for most encoders, unlike the synthetic GMM examples.
Limitations
- Theory is limited to linear models in a two-class Gaussian mixture setting.
- Empirical analysis assumes pretrained frozen encoders; mechanism by which neural feature learning creates spectral imbalance is not studied.
- Spectral imbalance is only one of several possible factors for class disparity.
Experimental setup
Datasets: ImageNet ILSVRC 2012
Benchmarks: ImageNet ILSVRC 2012 validation set
Metrics: per-class probability of error class gap Pearson correlation coefficient Spectral Quantile Score top-1 accuracy Welch's t-test
Baselines: ResNet-50 ViT-B DenseNet EfficientNet MLP-Mixer PoolFormer
Builds on: Convex Gaussian Min-Max Theorem benign overfitting power law eigenspectrum analysis
Closest papers by meaning
- Learning under Localized Minority Imbalance (NeurIPS 2026 · similarity 0.90)
- Class-Attribute Priors: Adapting Optimization to Heterogeneity and Fairness Objective (AAAI 2024 · similarity 0.88)
- Classes Are Not Equal: An Empirical Study on Image Recognition Fairness (CVPR 2024 · similarity 0.87)
- Balancing the Scales: A Theoretical and Algorithmic Framework for Learning from Imbalanced Data (ICML 2025 · similarity 0.87)
- The Group Robustness is in the Details: Revisiting Finetuning under Spurious Correlations (NeurIPS 2024 · similarity 0.87)
- Twice Class Bias Correction for Imbalanced Semi-supervised Learning (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.