Research map / Learning theory and algorithms / Kernels and regression theory
Restoring balance: principled under/oversampling of data for optimal classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives replica-method asymptotics for linear SVMs on imbalanced one-hot (Potts) data in the high-dimensional limit and uses them to show that mixed under/oversampling restores balance and improves balanced accuracy, with RBM-based likelihood-informed sampling validated on real data.
Problem
Class imbalance degrades linear classifiers' generalization on minority classes, yet how under/oversampling should adapt to data statistics (imbalance, first and second moments, metric) is poorly understood theoretically.
Approach
Exact high-dimensional asymptotics (replica method) for spherical-perceptron/soft-margin SVM on categorical (Potts) data with class-dependent means and shared covariance, giving closed-form predictions for confusion-matrix-based metrics and for random under/oversampling mixtures; the theory predicts that mixed under+oversampling is optimal for balanced accuracy under severe imbalance, and motivate
Key results
- Theory matches simulations of linear SVMs on pMNIST, FMNIST and CelebA for balanced accuracy curves, with predictions averaged over 10 trials.
- Full random undersampling is found to be sub-optimal; for severe imbalance a mixed under/oversampling strategy gives highest balanced accuracy compared to full random oversampling.
- RBM-based likelihood-informed sampling on binary MNIST improves linear SVM performance over naive random under/oversampling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Restore class balance with a mixed under/oversampling strategy | Linear SVM · Imbalanced classification | balanced accuracy | Helped |
| Use full undersampling instead of mixed under/oversampling or oversampling | Linear SVM · Balanced training set | balanced accuracy | Hurt |
| Use mixed under/oversampling rather than full oversampling under severe imbalance | Linear SVM · Severe class imbalance | balanced accuracy | Helped |
| Use full oversampling rather than a mixed strategy when imbalance is milder | Linear SVM · Milder class imbalance | balanced accuracy | Helped |
| Use likelihood-informed sampling with an RBM rather than naive random under/oversampling | Linear SVM · MNIST (digits smaller/larger than 5) | performance | Helped |
| Restore class balance instead of training on imbalanced data | ImageNet-pretrained ResNet-50 · CIFAR-10 (labels smaller/larger than 5) | accuracy | Helped |
| Increase the SVM margin | Linear SVM · Synthetic data with varying training class composition | balanced accuracy | Mixed |
| Evaluate imbalanced training with accuracy rather than balanced accuracy | Linear classifier · Test set distributed as training | minority-class performance | Hurt |
| Use AUC as the performance measure rather than balanced accuracy under imbalance | Linear classifier · Imbalanced training | balanced accuracy | Hurt |
| Use the theory's mean empirical covariance when class covariance matrices differ | Linear SVM · Benchmark datasets including pMNIST, Fashion MNIST, and CelebA | agreement with numerical balanced-accuracy curves | Mixed |
Surprising findings
- AUC is largely insensitive to class imbalance and can be misleading relative to balanced accuracy.
- Full undersampling is sub-optimal among restoring-balance protocols in all tested cases.
Limitations
- Theory assumes homoscedastic classes (shared covariance) and convexity/strict convexity of the ERM problem.
- Real-data agreement is only qualitative for diverse covariances such as 0/1 MNIST.
- Deep-network experiments are only qualitative; theory is for linear models.
Experimental setup
Datasets: MNIST Fashion-MNIST CelebA CIFAR-10 Lattice Proteins
Benchmarks: MNIST Fashion-MNIST CelebA CIFAR-10
Metrics: balanced accuracy accuracy AUC AUPRC MCC F1
Baselines: random oversampling random undersampling SMOTE RBM-SMOTE imbalanced ERM
Builds on: replica method SMOTE RBM-SMOTE
Closest papers by meaning
- A Theoretical Analysis of the Learning Dynamics under Class Imbalance (ICML 2023 · similarity 0.89)
- Balancing the Scales: A Theoretical and Algorithmic Framework for Learning from Imbalanced Data (ICML 2025 · similarity 0.88)
- Concentration and excess risk bounds for imbalanced classification with synthetic oversampling (NeurIPS 2025 · similarity 0.88)
- Learning under Localized Minority Imbalance (NeurIPS 2026 · similarity 0.86)
- Balanced Data, Imbalanced Spectra: Unveiling Class Disparities with Spectral Imbalance (ICML 2024 · similarity 0.85)
- Improved Balanced Classification with Theoretically Grounded Loss Functions (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.