Research map / Label-efficient and robust learning / Recognition and long-tail
Measures of Information Reflect Memorization Patterns
Rachit Bansal, Danish Pruthi, Yonatan Belinkov
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase heuristic memorization strength alpha (up to 1.0) | MLP on Colored MNIST, DistilBERT on Sentiment Adjectives · synthetic spurious correlation | neuron entropy | Hurt |
| increase heuristic memorization strength alpha | MLP and DistilBERT · Colored MNIST, Sentiment Adjectives | neuron-pair mutual information | Hurt |
| train on original raw Bias-in-Bios set instead of balanced or scrubbed sets | RoBERTa-base · Bias-in-Bios | entropy | Hurt |
| de-biasing training sets (balanced, scrubbed) | RoBERTa-base · Bias-in-Bios | entropy vs compression | No clear effect |
| balanced instead of unbalanced common contexts in training | ResNet-18 · NICO++ | entropy: 35.03% -> 50.08% | Improved |
| unbalanced instead of balanced contexts in training | ResNet-18 · NICO++ | neuron-pair mutual information | Hurt |
| increase label shuffling fraction beta | 3-layer MLP on MNIST · Shuffled MNIST | entropy | Improved |
| increase label shuffling fraction beta | DistilBERT · Shuffled IMDb | entropy | Mixed |
| increase label shuffling fraction beta | MLP and DistilBERT · Shuffled MNIST, Shuffled IMDb | neuron-pair mutual information | Hurt |
| increase label shuffling fraction beta | MLP and DistilBERT · Shuffled MNIST, Shuffled IMDb | validation accuracy | Hurt |
| use mean entropy for model selection ranking | information measures · Bias-in-Bios TPR Gap | Kendall tau: 0.20 | No clear effect |
| use mean MI for model selection ranking | information measures · Colored MNIST | Kendall tau: 1.00 | Improved |
Closest papers by meaning
- A mathematical theory of balancing relational generalization and memorization (NeurIPS 2026 · similarity 0.89)
- Unveiling Memorization–Generalization Coexistence: A Case Study on Arithmetic Tasks with Label Noise (NeurIPS 2026 · similarity 0.88)
- The Pitfalls of Memorization: When Memorization Hurts Generalization (ICLR 2025 · similarity 0.88)
- Differential learning kinetics govern the transition from memorization to generalization during in-context learning (ICLR 2025 · similarity 0.88)
- Why Diffusion Models Don’t Memorize: The Role of Implicit Dynamical Regularization in Training (NeurIPS 2025 · similarity 0.87)
- Fisher information and the geometry of memorization in neural networks (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
