Research map / Learning theory and algorithms / Kernels and regression theory
Generalization on the Unseen, Logic Reasoning and Degree Curriculum
Emmanuel Abbe, Samy Bengio, Aryo Lotfi, Kevin Rizk
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add permutation-invariance regularizer (E[(f(x)-f(pi(x)))^2]) to Transformer loss (ablation) | encoder-only Transformer on f3 (cyclic-symmetric target) · f3 with unseen domain U3, input dim 15 | generalization error: 8.14 ± 0.9 -> 0.95 ± 0.13 | Improved |
| Use causal attention mask with no positional embeddings instead of bidirectional encoder (ablation) | Transformer on parity-2 with one frozen bit · parity of two bits, d=16, causal masking | leakage coefficient alpha_Leak: min-degree bias lost for some placements (e.g., alpha_Leak 0.55 for x0x1 with x0 frozen vs ~0 for bidirectional) | Hurt |
| Use causal attention mask with positional embeddings (ablation) | Transformer on parity-2 with one frozen bit · parity of two bits, d=16, causal masking | leakage coefficient alpha_Leak: only one placement with non-negligible leakage (0.17 for x0x1 with x0 frozen) | No clear effect |
| Random features model with strongly expressive (polynomial) activation vs ReLU activation | Random features model · f=x0x1x2+x0x3x4x5 with U={x0=-1}, input dim 15 | learned min-degree bias | Hurt |
| Train MLP with Degree-Curriculum (increasing Hamming-ball curriculum) instead of plain SGD | MLP learning full parity of degree 16 and 30 · parity 16 and 30 bits, varying sample size | generalization loss / sample complexity | Improved |
| Train on smaller Hamming-ball radius (leakier length-bounded training) for MLP parity learning | MLP learning full 15-bit parity · training on B_r with shorter lengths | coefficient of full parity term | Hurt |
| Increase ambient dimension for small-d regime (parity_2 / parity_4 embedded) | Transformer and RF on parity_2 and parity_4 · ambient dimension sweep | leakage coefficient alpha_Leak | Improved |
Closest papers by meaning
- Boule or Baguette? A Study on Task Topology, Length Generalization, and the Benefit of Reasoning Trace (NeurIPS 2026 · similarity 0.88)
- On the Minimal Degree Bias in Generalization on the Unseen for non-Boolean Functions (ICML 2024 · similarity 0.87)
- Grokking of Implicit Reasoning in Transformers: A Mechanistic Journey to the Edge of Generalization (NeurIPS 2024 · similarity 0.87)
- A Measure-Theoretic Analysis of Reasoning: Structural Generalization and Approximation Limits (NeurIPS 2026 · similarity 0.86)
- Improving Complex Reasoning over Knowledge Graph with Logic-Aware Curriculum Tuning (AAAI 2025 · similarity 0.86)
- Grokking in the Wild: Data Augmentation for Real-World Multi-Hop Reasoning with Transformers (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
