Research map / Probabilistic and scientific modeling / Variational and generative sampling
Adaptive Robust Learning using Latent Bernoulli Variables
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RLVI, a parameter-free robust learning method that marginalizes latent Bernoulli clean/corrupt indicators via variational EM under the Huber contamination model, and applies it to parameter estimation, online learning, and deep learning with label noise.
Problem
Maximum likelihood learning from corrupted training data where the corruption distribution and corruption level are unknown and vary (e.g., online streams), so existing robust methods need an upper bound or known noise level.
Approach
Introduce latent Bernoulli indicators for clean vs corrupted samples under the Huber contamination model, marginalize them via a variational bound, and optimize the corruption level in closed form (epsilon = 1 - mean(pi)), yielding sample weights pi_i solved by fixed-point EM iterations; for overparameterized networks, a type II error-based truncation threshold removes low-weight samples.
Key results
- On CIFAR10 with 45% symmetric noise, RLVI reached 88.69% test accuracy vs 83.33% for USDNL and 86.33% for JoCoR (mean over five runs).
- On MNIST with 45% pairflip noise, RLVI achieved 98.52% vs 98.22% for BARE, the strongest baseline in that setting.
- RLVI's per-epoch time on CIFAR100 was 10.52 s vs 8.82 s for standard SGD, indicating small overhead.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RLVI instead of standard likelihood maximization | ResNet18 · CIFAR10, synthetic label corruption | test accuracy | Helped |
| Use RLVI instead of standard likelihood maximization | ResNet34 · CIFAR100, synthetic label corruption | test accuracy | Helped |
| Use RLVI instead of standard likelihood maximization | LeNet · MNIST, synthetic label corruption | test accuracy | Helped |
| Add hard truncation of low non-corruption probabilities after overfitting begins (ablation) | RLVI neural-network training · CIFAR10, pairflip 45% noise | test accuracy | Helped |
| Omit hard truncation of low non-corruption probabilities (ablation) | RLVI neural-network training · CIFAR10 with synthetic label noise | test accuracy | Hurt |
| Use RLVI rather than SGD or incremental RRM | Online binary classification · HAR streaming batches with varying corruption | recall | Helped |
| Use RLVI rather than competing robust-learning methods | Synthetic linear regression, logistic regression, and PCA · Synthetic benchmark problems | model error | Helped |
| Use constrained RLVI to require at least n0 retained samples (ablation) | RLVI covariance estimation · Covariance estimation with unbounded likelihood | relative error | Helped |
Surprising findings
- Without regularization, RLVI's identified corrupted fraction decreases as the model fits corrupted labels, which is why truncation is needed for overparameterized networks.
- On Food101, RLVI performed best without regularization, which the authors attribute to weaker overfitting on that dataset.
Limitations
- Unbounded likelihoods (e.g., covariance estimation) can yield degenerate solutions requiring extra constraints.
- Overparameterized models overfit corrupted samples, requiring a truncation heuristic with type II error threshold; type II error not always below 5% in experiments.
- Alternative methods rely on the true or estimated corruption level, and RLVI's gains are not uniform (e.g., CIFAR10 asymmetric 45% where BARE is best).
Experimental setup
Datasets: MNIST CIFAR10 CIFAR100 CIFAR80N-O Food101 HAR (Human Activity Recognition) ImageNet
Benchmarks: MNIST CIFAR10 CIFAR100 CIFAR80N-O Food101 HAR online classification
Metrics: test accuracy relative error recall type I/II error time per epoch
Baselines: Standard SGD Huber Sever RRM Co-teaching JoCoR CDR USDNL BARE
Builds on: Huber contamination model Co-teaching Sever Robust Risk Minimization BARE
Closest papers by meaning
- Probabilistically Robust Learning: Balancing Average and Worst-case Performance (ICML 2022 · similarity 0.87)
- Robust Learning of Fixed-Structure Bayesian Networks (NeurIPS 2018 · similarity 0.86)
- Robust Learning of Fixed-Structure Bayesian Networks in Nearly-Linear Time (ICLR 2021 · similarity 0.86)
- Probably Approximately Global Robustness Certification (ICML 2025 · similarity 0.85)
- Enhancing Robust Fairness via Confusional Spectral Regularization (ICLR 2025 · similarity 0.85)
- Regularized least squares learning with heavy-tailed noise is minimax optimal (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.