Research map / Probabilistic and scientific modeling / Variational and generative sampling

Low-Dimension-to-High-Dimension Generalization and Its Implications for Length Generalization

ICML 2025 · Probabilistic and scientific modeling › Variational and generative sampling · approach: Tractable probabilistic models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Formalizes length generalization as low-dimension-to-high-dimension (LDHD) generalization, proves a no-free-lunch theorem, analyzes inductive biases of random feature models with projection and position-only linear attention under (S)GD, and proposes RPE-Square position embedding evaluated on unaligned copy and URF addition.

Problem

Length generalization in reasoning is confounded by data format nuisances and the mismatch between input length and true problem scale; the paper asks when scale shift from low to high dimensional latent spaces can be learned at all.

Approach

Model length generalization as LDHD generalization in a latent space of dimension equal to problem scale, separate from the data format mapping; show via no-free-lunch that inductive bias is necessary, and that architectures trained with GD converge to min-degree interpolators w.r.t. architecture-specific linearly independent sets, so LDHD succeeds only when the target aligns with that bias. Leads

Key results

• RPE-Square achieves out-of-distribution generalization on unaligned copy for lengths 6-10 when trained on lengths 1-5, while RPE only generalizes in-distribution.
• On URF addition trained on 4-addition and tested on 5-addition, RPE fails while RPE-Square succeeds in length generalization.
• Theory: RFM with projection and PLAA with GRPE converge to min-degree interpolators w.r.t. different linearly independent sets; PLAA with APE cannot achieve LDHD generalization when A*_ij nonzero for j>=i>N0.

Measured effects reported in the paper

Surprising findings

• Random feature models with projection, and PLAA with APE, do not share the min-degree bias of the standard RFM, showing that the min-degree profile bias does not hold for all architectures.
• CoT is argued to help by reshaping the latent space rather than changing its dimension, despite CoT producing more varied input lengths.

Limitations

• Theory uses simplified models (linear attention, random features) rather than full transformers.
• Experiments are limited to small synthetic tasks (unaligned copy and URF addition).
• The position embedding principle has not been turned into a general paradigm.

Experimental setup

Datasets: synthetic unaligned copy synthetic URF addition

Benchmarks: unaligned copy (lengths 1-10) URF addition

Metrics: accuracy digitwise accuracy

Baselines: RPE

Builds on: Abbe et al. 2023 min-degree interpolator relative position embedding chain-of-thought

Closest papers by meaning

• The Role of Sparsity for Length Generalization in LLMs (ICML 2025 · similarity 0.85)
• HIDISC: A Hyperbolic Framework for Domain Generalization with Generalized Category Discovery (NeurIPS 2025 · similarity 0.85)
• A Measure-Theoretic Analysis of Reasoning: Structural Generalization and Approximation Limits (NeurIPS 2026 · similarity 0.84)
• Sharpness-Aware Model-Agnostic Long-Tailed Domain Generalization (AAAI 2024 · similarity 0.84)
• On the Minimal Degree Bias in Generalization on the Unseen for non-Boolean Functions (ICML 2024 · similarity 0.84)
• How Many Domains Suffice for Domain Generalization? A Tight Characterization via the Domain Shattering Dimension (NeurIPS 2025 · similarity 0.84)