Research map / Label-efficient and robust learning / Recognition and long-tail
First-Order Manifold Data Augmentation for Regression Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FOMA, a domain-independent, data-driven augmentation for regression that scales down small singular values of batch representations (inputs/latents concatenated with labels) to sample from estimated tangent planes of the data manifold, and shows gains over ERM and mixup variants on in-distribution and out-of-distribution regression benchmarks.
Problem
Domain-independent data augmentation for deep regression is underexplored; mixup-style methods designed for classification are less effective for regression, and regularization of deep regression networks is not well understood.
Approach
Generate new samples by SVD of the concatenated [latent/input, label] batch matrix and scaling down its small singular values by a random lambda ~ Beta(alpha, alpha), so new samples lie near the tangent plane of the data manifold; the method is fully differentiable and applicable at any layer, and the loss is not rescaled.
Key results
- On Airfoil, FOMA_rho reaches RMSE 1.471 vs 2.901 for ERM and 2.357 for ADA (starred reproduction), roughly 37% below the prior state of the art.
- On Exchange-Rate, FOMA_rho achieves RMSE 0.013 vs 0.0236 for ERM, about 38% reduction over the best baseline.
- On out-of-distribution tasks, FOMA is mixed: it does not beat the best baselines on RCF (0.171 vs 0.153 for C-Mixup*) and PovertyMap, while FOMA_rho gives the best DTI average R of 0.503.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FOMA instead of ERM | regression models · Airfoil | RMSE: 2.901 -> 1.646 | Helped |
| Use FOMAρ instead of ERM | regression models · NO2 | RMSE: 0.537 -> 0.512 | Helped |
| Use FOMA instead of ERM | regression models · Exchange-Rate | RMSE: 0.0236 -> 0.013 | Helped |
| Use FOMA instead of ERM | regression models · Electricity | RMSE: 0.0581 -> 0.058 | No clear effect |
| Use FOMA instead of ERM | regression models · Echo | RMSE: 5.402 -> 5.215 | Helped |
| Use FOMAρ instead of ERM | regression models · DTI | R (average): 0.483 -> 0.503 | Helped |
| Use FOMAρ instead of ERM | regression models · PovertyMap | R (average): 0.80 -> 0.832 | Helped |
| Use FOMA instead of ERM | regression models · RCF | RMSE (average): 0.164 -> 0.171 | Hurt |
| Scale large rather than small singular values (ablation) | FOMA on NO2 · NO2 | RMSE: 0.52 ± 0.01 -> 0.79 ± 0.02 | Hurt |
| Apply FOMA in latent space instead of input space (ablation) | FOMA on Electricity · Electricity | RMSE: 5.83e^{-2} ± 1e^{-3} -> 6.11e^{-2} ± 9e^{-4} | Hurt |
| Apply FOMA in input and latent spaces (ablation) | FOMA on NO2 · NO2 | RMSE: 0.52 ± 0.01 -> 0.52 ± 0.01 | No clear effect |
| Scale the loss by λ² rather than leave it unscaled (ablation) | FOMA on NO2 · NO2 | RMSE: 0.52 ± 0.01 -> 0.54 ± 0.01 | Hurt |
Surprising findings
- Non-augmented models attain lowest test error on singular-value-perturbed test data at intermediate lambda (about 0.5), and the MSE on original data (lambda=1) is not the lowest, which the authors find unexpected for overfit tiny datasets.
- Reproduced baselines (C-Mixup, ADA) could not match their originally reported numbers, so the starred reproduced results are used for comparison.
Limitations
- SVD cost may be restrictive for large batches, and it is governed by batch size.
- Gains are mixed on out-of-distribution benchmarks, where FOMA does not beat the best baselines on RCF and PovertyMap.
- Hyperparameter choices (input vs latent, small vs large singular values, rho vs ID) are not consistently derivable across datasets.
Experimental setup
Datasets: Airfoil Self-Noise NO2 Exchange-Rate Electricity Echo RCFashionMNIST Communities and Crime DTI PovertyMap
Benchmarks: Airfoil NO2 Exchange-Rate Electricity Echo RCF-MNIST Crimes DTI PovertyMap
Metrics: RMSE MAPE R worst-domain RMSE worst-domain R
Baselines: ERM Mixup Manifold Mixup C-Mixup ADA ManiMixup additive noise
Builds on: mixup Manifold Mixup C-Mixup Anchor data augmentation TwoNN
Closest papers by meaning
- Curvature Enhanced Data Augmentation for Regression (ICML 2025 · similarity 0.89)
- Anchor Data Augmentation (NeurIPS 2023 · similarity 0.88)
- SF(DA)$^2$: Source-free Domain Adaptation Through the Lens of Data Augmentation (ICLR 2024 · similarity 0.86)
- AdvST: Revisiting Data Augmentations for Single Domain Generalization (AAAI 2024 · similarity 0.86)
- Characterizing the Generalization Error of Random Feature Regression with Arbitrary Data-Augmentation (NeurIPS 2026 · similarity 0.86)
- Tailoring Mixup to Data for Calibration (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.