Research map / Learning theory and algorithms / Kernels and regression theory

A mathematical theory of balancing relational generalization and memorization

NeurIPS 2026 · Learning theory and algorithms › Kernels and regression theory · approach: Statistical perspectives on generalization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces transitive inference with exceptions, analytically characterizes kernel ridge regression on it, and validates predictions by finetuning Llama3.2 and Qwen3.5 models on ordered relations with an intransitive loop.

Problem

How learning systems balance learning general relational rules (transitive generalization) with memorizing exceptions that create intransitive loops; existing paradigms (transitive inference, transverse patterning) study these only in isolation.

Approach

A new task, transitive inference with exceptions, embeds an intransitive loop in an otherwise transitive ordering; an exact analytical solution for kernel ridge regression with exchangeable kernels shows the emergent ranking becomes non-monotonic over the loop, which makes cross-section generalization geometry-dependent and produces systematic errors.

Key results

• Kernel model theory shows within-section generalization is always successful for alpha>0, while cross-section generalization can fail for small regularization-scaled values, with systematic errors predicting I_{q-1}<I_{p+1}.
• Finetuned PLMs (Llama3.2, Qwen3.5) generalize well within sections on TI with exceptions but show below-chance accuracy on cross-section generalization and some intransitive pairs for some models.
• Increasing the number of transitive items yields a stronger performance drop than increasing intransitive items in finetuned PLMs, as predicted by theory.

Surprising findings

• L2 regularization, which aids in-distribution generalization, can be counterproductive for out-of-distribution cross-section generalization and induce systematic errors.
• Kernel models never make systematic errors on standard transitive inference or transverse patterning, but do on the exception task.

Limitations

• Analysis limited to kernel models with exchangeable representations, not feature-learning networks.
• Only one specific form of compositional generalization (relational ordering) studied.
• Hyperparameters chosen for default behavior rather than optimized; LoRA results not well captured by theory.

Experimental setup

Datasets: synthetic transitive inference hierarchies with sports teams (fake and real MLB names) Texas Hold'em hand win-probability hierarchies (eval7 equity matrix)

Benchmarks: transitive inference with exceptions (own task) transitive inference (n=9)

Metrics: accuracy margin

Baselines: transverse patterning transitive inference

Builds on: neural tangent kernel transitive inference transverse patterning LoRA

Closest papers by meaning

• The Pitfalls of Memorization: When Memorization Hurts Generalization (ICLR 2025 · similarity 0.90)
• Unveiling Memorization–Generalization Coexistence: A Case Study on Arithmetic Tasks with Label Noise (NeurIPS 2026 · similarity 0.89)
• Generalization v.s. Memorization: Tracing Language Models’ Capabilities Back to Pretraining Data (ICLR 2025 · similarity 0.87)
• To Grok or not to Grok: Disentangling Generalization and Memorization on Corrupted Algorithmic Datasets (ICLR 2024 · similarity 0.87)
• What Do Learning Dynamics Reveal About Generalization in LLM Mathematical Reasoning? (ICML 2025 · similarity 0.87)
• Differential learning kinetics govern the transition from memorization to generalization during in-context learning (ICLR 2025 · similarity 0.87)