Research map / Label-efficient and robust learning / Recognition and long-tail
Learn2Mix: Training Neural Networks Using Adaptive Data Integration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes learn2mix, which adaptively reweights class proportions within each training batch toward classes with higher error, and reports faster convergence than classical, focal-loss, SMOTE, importance-sampling and curriculum baselines on classification, regression and reconstruction tasks.
Problem
Standard training keeps class proportions fixed within batches, which ignores differing class difficulty and can slow convergence, especially under class imbalance and limited training budgets.
Approach
Update batch class proportions each epoch by nudging mixing parameters toward the normalized vector of class-wise losses (weighted by a mixing rate gamma), so harder classes get more samples per batch; the paper proves convergence of parameters and mixing proportions under strong convexity and Lipschitz assumptions.</ättr>
Key results
- On MNIST at the full budget, learn2mix reaches 98.46% test accuracy vs 98.14% for classical training; at 0.25E on CIFAR-10 it reaches 51.34% vs 49.26% for classical.
- On Imagenette learn2mix reaches 40% test accuracy after 185 s, whereas only the FCL-trained model reaches it by 230 s.
- On Wine Quality at epoch 0.25E learn2mix MSE is 17.7 vs 19.8 for classical, and at epoch E it is 1.75 vs 2.03.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Adapt batch class proportions using class-wise losses | Learn2Mix versus classical training · MNIST classification, elapsed time = E | accuracy: 98.14 \scriptstyle \pm 0.14 -> 98.46 \scriptstyle \pm 0.14 | Helped |
| Adapt batch class proportions using class-wise losses | Learn2Mix versus classical training · Fashion-MNIST classification, elapsed time = E | accuracy: 84.23 \scriptstyle \pm 0.60 -> 85.85 \scriptstyle \pm 0.47 | Helped |
| Adapt batch class proportions using class-wise losses | Learn2Mix versus classical training · CIFAR-10 classification, elapsed time = E | accuracy: 59.62 \scriptstyle \pm 0.16 -> 60.49 \scriptstyle \pm 0.26 | Helped |
| Adapt batch class proportions using class-wise losses | Learn2Mix versus classical training · Imagenette classification, elapsed time = E | accuracy: 34.53 \scriptstyle \pm 0.33 -> 42.95 \scriptstyle \pm 0.33 | Helped |
| Adapt batch class proportions using class-wise losses | Learn2Mix versus classical training · CIFAR-100 classification, elapsed time = E | accuracy: 52.30 \scriptstyle \pm 0.36 -> 54.50 \scriptstyle \pm 0.73 | Helped |
| Adapt batch class proportions using class-wise losses | Learn2Mix versus classical training · IMDB classification, elapsed time = E | accuracy: 80.84 \scriptstyle \pm 0.71 -> 82.38 \scriptstyle \pm 0.59 | Helped |
| Adapt batch proportions using class-wise losses | Learn2Mix versus classical training · Mean Estimation regression, epoch = E | MSE: 1.17 \scriptstyle \pm 0.06 -> 1.07 \scriptstyle \pm 0.09 | Helped |
| Adapt batch proportions using class-wise losses | Learn2Mix versus classical training · Wine Quality regression, epoch = E | MSE: 2.03 \scriptstyle \pm 0.18 -> 1.75 \scriptstyle \pm 0.21 | Helped |
| Adapt batch proportions using class-wise losses | Learn2Mix versus classical training · California Housing regression, epoch = E | MSE: 0.99 \scriptstyle \pm 0.10 -> 0.77 \scriptstyle \pm 0.08 | Helped |
| Adapt batch proportions using class-wise losses | Learn2Mix versus classical training · MNIST reconstruction, epoch = E | MSE: 10.1 \scriptstyle \pm 0.56 -> 9.31 \scriptstyle \pm 0.24 | Helped |
| Adapt batch proportions using class-wise losses | Learn2Mix versus classical training · Fashion-MNIST reconstruction, epoch = E | MSE: 51.6 \scriptstyle \pm 1.60 -> 45.5 \scriptstyle \pm 1.21 | Helped |
| Adapt batch proportions using class-wise losses | Learn2Mix versus classical training · CIFAR-10 reconstruction, epoch = E | MSE: 148 \scriptstyle \pm 1.37 -> 144 \scriptstyle \pm 1.71 | Helped |
Surprising findings
- On IMDB at 0.25E, the importance sampling baseline (58.48%) beats learn2mix (70.28% is higher, so not surprising); instead, SMOTE and FCL underperform classical training on several datasets at early budgets.
Limitations
- Convergence guarantees rely on strong convexity and Lipschitz assumptions that do not hold for deep networks.
- Requires choosing the mixing rate gamma; the paper suggests gamma in [0.01, 0.1] works well.
- Imbalanced settings are manually constructed by subsampling classes rather than naturally occurring imbalance.
Experimental setup
Datasets: MNIST Fashion-MNIST CIFAR-10 Imagenette CIFAR-100 IMDB Wine Quality California Housing Mean Estimation (synthetic)
Benchmarks: MNIST Fashion-MNIST CIFAR-10 Imagenette CIFAR-100 IMDB Wine Quality California Housing Mean Estimation
Metrics: test accuracy test mean squared error classification error vs elapsed time
Baselines: classical training FCL (focal loss) SMOTE training IS (importance sampling) training CURR (curriculum learning) training
Builds on: curriculum learning focal loss SMOTE importance sampling
Closest papers by meaning
- A Second-Order Perspective on Model Compositionality and Incremental Learning (ICLR 2025 · similarity 0.84)
- Accelerated Training via Incrementally Growing Neural Networks using Variance Transfer and Learning Rate Adaptation (NeurIPS 2023 · similarity 0.84)
- How Gradient descent balances features: A dynamical analysis for two-layer neural networks (ICLR 2025 · similarity 0.84)
- Asymptotic Analysis of Two-Layer Neural Networks after One Gradient Step under Gaussian Mixtures Data with Structure (ICLR 2025 · similarity 0.83)
- RegMix: Data Mixture as Regression for Language Model Pre-training (ICLR 2025 · similarity 0.83)
- On the Parameterization of Second-Order Optimization Effective towards the Infinite Width (ICLR 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.