Research map / Label-efficient and robust learning / Noisy labels

Unveiling Memorization–Generalization Coexistence: A Case Study on Arithmetic Tasks with Label Noise

NeurIPS 2026 · Label-efficient and robust learning › Noisy labels · approach: Noisy cross-modal hashing and multi-label · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies how over-parameterized two-layer networks coexist with memorizing heavy label noise on modular arithmetic, showing generalizing Fourier structure persists and can be extracted by frequency filtration, while neuron pruning helps only modestly.

Problem

How over-parameterized models can simultaneously memorize noisy labels and generalize well, and whether the generalizing rule is recoverable under heavy label noise, using modular arithmetic as a controlled testbed.

Approach

Uses the dominant Fourier frequency of each ReLU neuron as the generalization component (frequency filtration) and separates the rest as memorization; also proposes a task-agnostic neuron-strength score for neuron-level pruning into generalization and memorization subnetworks.

Key results

• Frequency filtration of the dominant-frequency sub-network recovers near-perfect test accuracy even with 80% label noise on modular addition.
• Noisy labels are memorized faster than clean labels early in full-batch training across ReLU networks, under AdamW, Adam, and Muon.
• Neuron selection by task-agnostic neuron strength gives only limited generalization gains compared with frequency filtration applied to the full model.
• On a Transformer with label noise, frequency filtration keeping top 20 frequencies raised test accuracy from 76.8% to 87.8% and reduced noisy-label accuracy from 100% to 36.8%.

Surprising findings

• Noisy labels are memorized faster than clean labels at early training under full-batch optimization.
• Larger models generalize better under noise in the over-parameterized regime, with double descent in test error.

Limitations

• Setting is a controlled algebraic task that abstracts away natural vision or language variability.
• Frequency filtration relies on prior knowledge of task-specific Fourier structure and does not give a strict functional decomposition.
• Tied-weight models cannot interpolate asymmetric noise, so over-parameterized regime may not exist.

Experimental setup

Datasets: modular addition (P=113) modular subtraction modular multiplication MNIST

Benchmarks: modular addition modular subtraction modular multiplication MNIST

Metrics: test accuracy clean accuracy noisy-label accuracy test cross-entropy inverse participation ratio phase MSE

Baselines: Adam AdamW Muon SGD quadratic activation network IPR-based neuron selection

Builds on: grokking double descent analytical solution for quadratic-activation networks (margin maximization) Muon optimizer

Closest papers by meaning

• To Grok or not to Grok: Disentangling Generalization and Memorization on Corrupted Algorithmic Datasets (ICLR 2024 · similarity 0.90)
• The Pitfalls of Memorization: When Memorization Hurts Generalization (ICLR 2025 · similarity 0.89)
• A mathematical theory of balancing relational generalization and memorization (NeurIPS 2026 · similarity 0.89)
• Fisher information and the geometry of memorization in neural networks (NeurIPS 2026 · similarity 0.89)
• Generalizablity of Memorization Neural Network (NeurIPS 2024 · similarity 0.88)
• The Geometric Inductive Bias of Grokking: Bypassing Phase Transitions via Architectural Topology (NeurIPS 2026 · similarity 0.88)