Research map / Label-efficient and robust learning / OOD and anomaly detection
The Pitfalls of Memorization: When Memorization Hurts Generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that spurious correlations combined with memorization hurt ERM generalization, and proposes memorization-aware training (MAT), which shifts logits by calibrated held-out predictions from an XRM auxiliary model.
Problem
Neural networks trained with ERM can rely on spurious correlations for majority examples while memorizing minority exceptions, reaching zero training loss yet failing on minority groups under subpopulation shift.
Approach
Formalizes how example-specific features enable memorization that masks reliance on spurious features, and introduces MAT, which adds a per-example log of calibrated held-out (XRM-derived) probabilities to the logits during training so that the loss prioritizes hard, poorly generalized examples.
Key results
- On CelebA with no group annotations in validation, MAT reaches 89.9% worst-group accuracy versus 54.3% for ERM (and 89.1% for XRM).
- On Waterbirds with validation group annotations, MAT achieves 88.1% worst-group accuracy versus 72.6% for ERM.
- MAT produces more uniform self-influence distributions across majority and minority groups than ERM, indicating reduced memorization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MAT instead of ERM | ResNet-50 · Waterbirds (validation group annotations) | worst-group test accuracy: 72.6 -> 88.1 | Helped |
| Use MAT instead of ERM | ResNet-50 · CelebA (validation group annotations) | worst-group test accuracy: 47.2 -> 90.5 | Helped |
| Use MAT instead of ERM | BERT · MultiNLI (validation group annotations) | worst-group test accuracy: 67.9 -> 74.6 | Helped |
| Use MAT instead of ERM | BERT · CivilComments (validation group annotations) | worst-group test accuracy: 69.5 -> 74.0 | Helped |
| Use MAT instead of ERM | ResNet-50 · Waterbirds (no validation group annotations) | worst-group test accuracy: 66.4 -> 88.1 | Helped |
| Use MAT instead of ERM | ResNet-50 · CelebA (no validation group annotations) | worst-group test accuracy: 54.3 -> 89.9 | Helped |
| Use MAT instead of ERM | BERT · MultiNLI (no validation group annotations) | worst-group test accuracy: 67.9 -> 73.0 | Helped |
| Use MAT instead of ERM | BERT · CivilComments (no validation group annotations) | worst-group test accuracy: 67.2 -> 68.1 | Mixed |
| Use MAT instead of XRM | BERT · CivilComments (no validation group annotations) | worst-group test accuracy: 72.2 -> 68.1 | Hurt |
Surprising findings
- Spurious correlations alone do not necessarily cause poor generalization; the combination with memorization does.
- Memorization can be benign or even beneficial in some regimes (label noise with example-specific features), so memorization is not always harmful.
Limitations
- MAT requires an auxiliary XRM model and a temperature hyperparameter tau, adding training cost.
- Model selection still relies on validation worst-group accuracy, which may use inferred or ground-truth group annotations.
- Theoretical analysis is limited to linear models with a specific Gaussian data setup.
Experimental setup
Datasets: Waterbirds CelebA MultiNLI CivilComments
Benchmarks: Waterbirds CelebA MultiNLI CivilComments
Metrics: average accuracy worst-group accuracy self-influence score
Baselines: ERM GroupDRO LfF JTT LC uLA AFR XRM+GroupDRO
Builds on: Cross-Risk Minimization (XRM) GroupDRO Learning from Failure (LfF) Just Train Twice (JTT) TRAK
Closest papers by meaning
- A mathematical theory of balancing relational generalization and memorization (NeurIPS 2026 · similarity 0.90)
- To Grok or not to Grok: Disentangling Generalization and Memorization on Corrupted Algorithmic Datasets (ICLR 2024 · similarity 0.89)
- Unveiling Memorization–Generalization Coexistence: A Case Study on Arithmetic Tasks with Label Noise (NeurIPS 2026 · similarity 0.89)
- How Spurious Features are Memorized: Precise Analysis for Random and NTK Features (ICML 2024 · similarity 0.89)
- Towards Memorization Estimation: Fast, Formal and Free (ICML 2025 · similarity 0.88)
- Memorization in Self-Supervised Learning Improves Downstream Generalization (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.