AI Paper Atlas

Research map / Learning theory and algorithms / Kernels and regression theory

Generalization on the Unseen, Logic Reasoning and Degree Curriculum

Emmanuel Abbe, Samy Bengio, Aryo Lotfi, Kevin Rizk

ICML 2023 · Learning theory and algorithms › Kernels and regression theory · approach: Statistical perspectives on generalization · 3 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Add permutation-invariance regularizer (E[(f(x)-f(pi(x)))^2]) to Transformer loss (ablation)encoder-only Transformer on f3 (cyclic-symmetric target) · f3 with unseen domain U3, input dim 15generalization error: 8.14 ± 0.9 -> 0.95 ± 0.13Improved
Use causal attention mask with no positional embeddings instead of bidirectional encoder (ablation)Transformer on parity-2 with one frozen bit · parity of two bits, d=16, causal maskingleakage coefficient alpha_Leak: min-degree bias lost for some placements (e.g., alpha_Leak 0.55 for x0x1 with x0 frozen vs ~0 for bidirectional)Hurt
Use causal attention mask with positional embeddings (ablation)Transformer on parity-2 with one frozen bit · parity of two bits, d=16, causal maskingleakage coefficient alpha_Leak: only one placement with non-negligible leakage (0.17 for x0x1 with x0 frozen)No clear effect
Random features model with strongly expressive (polynomial) activation vs ReLU activationRandom features model · f=x0x1x2+x0x3x4x5 with U={x0=-1}, input dim 15learned min-degree biasHurt
Train MLP with Degree-Curriculum (increasing Hamming-ball curriculum) instead of plain SGDMLP learning full parity of degree 16 and 30 · parity 16 and 30 bits, varying sample sizegeneralization loss / sample complexityImproved
Train on smaller Hamming-ball radius (leakier length-bounded training) for MLP parity learningMLP learning full 15-bit parity · training on B_r with shorter lengthscoefficient of full parity termHurt
Increase ambient dimension for small-d regime (parity_2 / parity_4 embedded)Transformer and RF on parity_2 and parity_4 · ambient dimension sweepleakage coefficient alpha_LeakImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy