Research map / Learning theory and algorithms / Kernels and regression theory

Characterizing the Generalization Error of Random Feature Regression with Arbitrary Data-Augmentation

NeurIPS 2026 · Learning theory and algorithms › Kernels and regression theory · approach: High-dimensional linear regression theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Derives deterministic equivalents for the generalization error of ridge-regularized random feature regression trained with arbitrary (possibly biased, dependent) data augmentation in the proportional regime, with non-asymptotic error bounds and MNIST/synthetic validation.

Problem

Generalization error of random-feature ridge regression trained with data augmentation is not characterized by existing theory, which mostly covers i.i.d. data and unbiased, label-preserving augmentations; augmentations induce dependence across samples and can be biased.

Approach

Extends random matrix deterministic equivalents to a dependent sample setting where each (feature, augmented-feature) pair is jointly distributed, allowing biased augmentations and label-modifying augmentations under feature misspecification, via a leave-two-out argument and dilation fixed-point system.

Key results

• Deterministic equivalent for the augmented ridge generalization error matches Monte-Carlo empirical error on synthetic Gaussian data across lambda and alpha, with remainder bound scaling as lambda^{-9/2} n^{-1/2}.
• On MNIST inpainting with identity feature map, all three augmentation schemes reduce test error mainly through variance reduction, with the deterministic equivalents nearly indistinguishable from empirical curves.
• Under feature misspecification, augmentation did not systematically increase bias for practitioner-relevant strengths, so gains come mostly from variance reduction.

Surprising findings

• Under feature misspecification, data augmentation does not systematically increase bias, contrary to the usual bias-variance intuition from well-specified settings.
• Augmentation can improve generalization over a broad range of ridge parameters without a monotonic bias increase.

Limitations

• Approximation deteriorates as the ridge parameter lambda goes to 0, so analysis restricted to lambda >= 1e-2 in experiments.
• Results do not directly give a practical criterion for when augmentation helps in finite samples.
• Stability (Lipschitz) assumptions on augmentation statistics are needed; only relaxed in future work.

Experimental setup

Datasets: MNIST synthetic Gaussian regression data

Benchmarks: MNIST inpainting (5x5 center patch) synthetic Gaussian regression

Metrics: mean squared error bias^2 variance

Baselines: non-augmented ridge regression

Builds on: random feature models Hanson-Wright inequality leave-one-out argument for sample covariance resolvents

Closest papers by meaning

• How Much Data Are Augmentations Worth? An Investigation into Scaling Laws, Invariance, and Implicit Regularization (ICLR 2023 · similarity 0.88)
• On the Inherent Regularization Effects of Noise Injection During Training (ICML 2021 · similarity 0.88)
• First-Order Manifold Data Augmentation for Regression Learning (ICML 2024 · similarity 0.86)
• PEER Pressure: Model-to-Model Regularization for Single Source Domain Generalization (CVPR 2025 · similarity 0.86)
• Spurious Correlations in High Dimensional Regression: The Roles of Regularization, Simplicity Bias and Over-Parameterization (ICML 2025 · similarity 0.85)
• Non-Asymptotic Analysis Of Data Augmentation For Precision Matrix Estimation (NeurIPS 2025 · similarity 0.85)