Research map / Label-efficient and robust learning / Recognition and long-tail
Memorization Through the Lens of Curvature of Loss Function Around Samples
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes averaging input-space loss curvature (Hutchinson estimate of trace of squared Hessian w.r.t. inputs) over training epochs as a single-run memorization score, validated against FZ memorization scores and synthetic label corruption.
Problem
Measuring how much a neural network memorizes individual training samples is costly (leave-one-out style baselines need thousands of models), and existing methods miss failure modes such as duplicated images with conflicting labels.
Approach
Use the curvature of the cross-entropy loss with respect to the input (trace of squared input Hessian, estimated with Rademacher probes and finite differences) averaged over training epochs as a per-sample memorization score, computed from a single training run with no extra models.
Key results
- Curvature scores reach cosine similarity 0.82 (CIFAR100) and 0.72 (ImageNet) with FZ scores under weight decay 1e-4, rising to 0.90 and 0.87 on top-K most memorized samples.
- Curvature achieves AUROC of 100% on MNIST, 90.9%-97.4% at 1% corruption on CIFAR10/CIFAR100 and above 92.9% on all datasets except CIFAR100 (above 84.3%) across corruption rates.
- Without weight decay, 36 of top 60 CIFAR100 and 45 of top 100 ImageNet highest-curvature samples are duplicated images with conflicting labels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use curvature instead of Confident Learning to rank synthetically corrupted samples | ResNet18 curvature scoring · MNIST, 1–10% label corruption | AUROC: 99.7% -> 100.0%; 99.3% -> 100.0%; 99.1% -> 99.9%; 98.9% -> 99.9%; 98.9% -> 99.9%; 98.9% -> 99.9% | Helped |
| Use curvature instead of Confident Learning to rank synthetically corrupted samples | ResNet18 curvature scoring · CIFAR10, 1–8% label corruption | AUROC: 92.4% -> 97.4%; 93.1% -> 96.6%; 93.4% -> 95.5%; 93.4% -> 94.4%; 93.5% -> 94.1% | Helped |
| Use curvature instead of Confident Learning to rank synthetically corrupted samples | ResNet18 curvature scoring · CIFAR10, 10% label corruption | AUROC: 93.4% -> 92.9% | Hurt |
| Use curvature instead of Confident Learning to rank synthetically corrupted samples | ResNet18 curvature scoring · CIFAR100, 1–8% label corruption | AUROC: 84.2% -> 89.6%; 85.3% -> 88.3%; 86.3% -> 86.8%; 84.3% -> 85.7% | Helped |
| Use curvature instead of Confident Learning to rank synthetically corrupted samples | ResNet18 curvature scoring · CIFAR100, 10% label corruption | AUROC: 84.6% -> 84.3% | Hurt |
| Add weight decay (1e-04) to curvature scoring (ablation) | ResNet18 curvature scoring · CIFAR100 | cosine similarity with FZ scores: 0.73 -> 0.82 | Helped |
| Add weight decay (1e-04) to curvature scoring (ablation) | ResNet18 curvature scoring · ImageNet | cosine similarity with FZ scores: 0.66 -> 0.72 | Helped |
Surprising findings
- Per-epoch curvature rises before overfitting and then decreases during prolonged training as cross-entropy margin maximization saturates softmax, motivating epoch averaging.
- Cosine similarity with FZ scores drops about 0.1 without weight decay, partly because the method catches duplicate-with-conflicting-label failure modes that FZ scores miss.
Limitations
- Curvature is not claimed to be the best mislabel detector; it underperforms Confident Learning on some CIFAR100 settings.
- Cosine similarity with FZ scores drops without weight decay, and per-epoch curvature trends are non-monotone during overfitting.
- Duplicates with identical labels are not identified by the method.
Experimental setup
Datasets: MNIST FashionMNIST CIFAR10 CIFAR100 ImageNet
Benchmarks: CIFAR100 FZ memorization scores ImageNet FZ memorization scores synthetic label corruption on MNIST, CIFAR10, CIFAR100
Metrics: cosine similarity AUROC
Baselines: FZ memorization scores inconfidence Confident Learning (cleanlab)
Builds on: Feldman and Zhang memorization scores (influence memorization) Confident Learning Hutchinson trace estimator
Closest papers by meaning
- Unveiling Privacy, Memorization, and Input Curvature Links (ICML 2024 · similarity 0.90)
- Localizing Memorized Regions in Diffusion Models via Coordinate-Wise Curvature Differences (ICML 2026 · similarity 0.90)
- Towards Memorization Estimation: Fast, Formal and Free (ICML 2025 · similarity 0.89)
- A Geometric Framework for Understanding Memorization in Generative Models (ICLR 2025 · similarity 0.87)
- Fisher information and the geometry of memorization in neural networks (NeurIPS 2026 · similarity 0.86)
- How Spurious Features are Memorized: Precise Analysis for Random and NTK Features (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.