Research map / Label-efficient and robust learning / OOD and anomaly detection

Does Your Neural Network Extrapolate? Feature Engineering as Identifiability Bias for OOD Generalization

NeurIPS 2026 · Label-efficient and robust learning › OOD and anomaly detection · approach: Out-of-distribution generalization theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Argues OOD extrapolation from a single training window is non-identifiable without a structural commitment (feature map, label map, model class), and shows via toy, cross-domain (chemistry, Kepler, DNA) and positional-encoding experiments that correct feature maps yield large OOD gains at equal ID loss.

Problem

When can neural networks extrapolate out-of-distribution from an in-distribution training window, given that many data-generating processes are observationally equivalent on the training data but diverge outside it?

Approach

Frames the feature map, label map and model class (phi, psi, M) as a joint DGP commitment that resolves OOD non-identifiability; correctness is relative to the model class; shows that correct features with a matching linear head give zero OOD error under realisability, and proposes near-boundary validation and a SINDy-based delta_OOD diagnostic for selecting the commitment.

Key results

• Fourier-feature MLP vs raw MLP at identical training error: about 0.3% vs 155.9% OOD relative error on sin(x) (~520x gap).
• Mass-action kinetics: bilinear-feature OLS reaches 0.0% OOD vs 24.4% for raw-input MLP.
• Kepler's law on NASA Exoplanet Archive: log-feature OLS 3.8% OOD (SINDy 2.1%) vs 79.0% for raw-feature MLP.

Surprising findings

• Identical pretrained foundation models (TimesFM, TabPFN) swing from near-chance to near-perfect OOD error solely by changing the input format.
• Correct feature map is necessary but not sufficient; wrong model class or insufficient coverage causes OOD failure even with correct features.

Limitations

• Information-theoretic bound is vacuous in the signal-dominated regime where most experiments sit.
• SINDy delta_OOD diagnostic is silent on non-polynomial dynamics and cannot rank candidates.
• Near-boundary validation requires accessible boundary samples; cross-species CDS head fails on high-GC organisms.

Experimental setup

Datasets: NASA Exoplanet Archive S. cerevisiae chromosomes I-IX E. coli B. subtilis S. pombe P. falciparum M. tuberculosis

Benchmarks: sin(x) vs Taylor-9 OOD extrapolation power law vs exponential mass-action kinetics Kepler's third law cross-species CDS detection periodic positional-encoding task

Metrics: OOD relative error (MAPE) ID error AUROC delta_OOD SINDy residual Spearman correlation

Baselines: MLP TabPFN TimesFM SINDy DNABERT Mamba S4D

Builds on: SINDy Rashomon set Fourier neural operator PINN

Closest papers by meaning

• Controlling Neural Collapse Enhances Out-of-Distribution Detection and Transfer Learning (ICML 2025 · similarity 0.85)
• What Variables Affect Out-of-Distribution Generalization in Pretrained Models? (NeurIPS 2024 · similarity 0.85)
• Detecting Out-of-Distribution Through the Lens of Neural Collapse (CVPR 2025 · similarity 0.84)
• Deep Neural Networks Tend To Extrapolate Predictably (ICLR 2024 · similarity 0.84)
• Position: Understanding LLMs Requires More Than Statistical Generalization (ICML 2024 · similarity 0.84)
• Mysteries of the Deep: Role of Intermediate Representations in Out of Distribution Detection (NeurIPS 2025 · similarity 0.84)