Research map / Learning theory and algorithms / Prediction and decision losses
Towards Memorization Estimation: Fast, Formal and Free
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Cumulative Sample Loss (CSL), a zero-overhead proxy for memorization derived from per-sample loss summed over training, with theory linking it to learning time and stability-based memorization.
Problem
Stability-based memorization scores are accurate but require training thousands of models, making them impractical; existing cheaper proxies lack a formal foundation linking learning dynamics to memorization.
Approach
Use the cumulative per-sample training loss over all SGD iterations as a memorization proxy, and prove that its expectation bounds both learning time and stability-based memorization, with the proxy obtainable at no extra training cost.
Key results
- CSL achieves cosine similarity 0.87 with stability-based memorization across all CIFAR-100 examples (Inception), versus 0.69 for input loss curvature.
- On ImageNet (ResNet50), CSL reaches cosine similarity 0.94 on the top 50k most memorized examples, versus 0.84 for curvature.
- For mislabeled detection on CIFAR-100 at 10% noise, CSL achieves AUROC 0.9897, and for duplicate detection on CIFAR-100 it achieves 0.9886 versus 0.9639 for curvature.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CSL instead of Curv. for mislabeled-example detection | mislabeled detection · CIFAR-10, 1%, 2%, 5%, and 10% symmetric label noise | AUROC: 0.9715 ± 0.0045 -> 0.9845 ± 0.0026; 0.9776 ± 0.0033 -> 0.9864 ± 0.0004; 0.9800 ± 0.0003 -> 0.9870 ± 0.0003; 0.9819 ± 0.0 | Helped |
| use CSL instead of Curv. for mislabeled-example detection | mislabeled detection · CIFAR-100, 1%, 2%, 5%, and 10% symmetric label noise | AUROC: 0.9856 ± 0.0009 -> 0.9891 ± 0.0003; 0.9865 ± 0.0011 -> 0.9895 ± 0.0002; 0.9876 ± 0.0021 -> 0.9895 ± 0.0001; 0.9892 ± 0.0 | Helped |
| use CSL instead of Curv. for duplicate detection | duplicate detection · CIFAR-10 | AUROC: 0.9536 ± 0.0030 -> 0.9821 ± 0.0006 | Helped |
| use CSL instead of Curv. for duplicate detection | duplicate detection · CIFAR-100 | AUROC: 0.9639 ± 0.0030 -> 0.9886 ± 0.0008 | Helped |
| use CSL instead of Loss Sensitivity as the memorization proxy | memorization estimation · CIFAR-100, all examples | cosine similarity: 0.81 -> 0.87 | Helped |
| use CSL instead of Loss Sensitivity as the memorization proxy | memorization estimation · CIFAR-100, Top 5k | cosine similarity: 0.97 -> 0.93 | Hurt |
| use CSL instead of Curv. as the memorization proxy | memorization estimation · ImageNet, all examples | cosine similarity: 0.62 -> 0.79 | Helped |
| use CSL instead of Curv. as the memorization proxy | memorization estimation · ImageNet, Top 50k | cosine similarity: 0.84 -> 0.94 | Helped |
Surprising findings
- Threshold-based learning time performs near chance (AUROC about 0.5) for mislabeled detection, indicating threshold-based proxies are ineffective.
Limitations
- Theoretical bounds rely on assumptions such as uniformly bounded loss, ρ-Lipschitz gradients, and bounded gradient variance, which cross-entropy loss only approximately satisfies.
- Stability-based memorization scores are precomputed from Feldman & Zhang (2020) and used as ground truth, so validation is limited to vision classification benchmarks.
- Learning-time and memorization relationships are validated only in binned, expectation-level analyses rather than per-sample.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet
Benchmarks: CIFAR-10 CIFAR-100 ImageNet
Metrics: cosine similarity Pearson correlation AUROC
Baselines: Threshold learning time In-confidence Confident Learning SSFT Input loss curvature Loss sensitivity Forgetting frequency Final sample loss
Builds on: stability-based memorization (Feldman 2020) input loss curvature (Garg et al. 2024) confident learning SSFT
Closest papers by meaning
- Memorization Through the Lens of Curvature of Loss Function Around Samples (ICML 2024 · similarity 0.89)
- Learning the Signature of Memorization in Autoregressive Language Models (NeurIPS 2026 · similarity 0.88)
- The Pitfalls of Memorization: When Memorization Hurts Generalization (ICLR 2025 · similarity 0.88)
- On the Memorization of Consistency Distillation for Diffusion Models (NeurIPS 2026 · similarity 0.88)
- Memorization in Self-Supervised Learning Improves Downstream Generalization (ICLR 2024 · similarity 0.88)
- Rethinking LLM Memorization through the Lens of Adversarial Compression (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.