Research map / Label-efficient and robust learning / Recognition and long-tail
APAR: Modeling Irregular Target Functions in Tabular Regression via Arithmetic-Aware Pre-Training and Adaptive-Regularized Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes APAR, a two-stage pretrain-finetune Transformer for tabular regression that pretrains by predicting arithmetic combinations of pairs of continuous labels and fine-tunes with a learned, correlated Bernoulli gate augmentation regularized by a consistency loss.
Problem
Deep learning models underperform GBDTs on tabular regression because irregular target functions (small feature changes causing large label changes) with continuous labels cause overfitting, and existing contrastive or masking methods either rely on discrete labels or corrupt important features.
Approach
An arithmetic-aware pretext task predicts the sum, difference, product, or quotient of two samples' continuous labels from their [CLS] representations, and a fine-tuning stage uses a learned gate vector (relaxed multivariate Bernoulli with Gaussian-copula correlation) to create augmented embeddings whose predictions must match the originals.
Key results
- APAR achieves the best average rank (1.3) across 10 datasets, outperforming the next best TabNet (3.8) in the main table.
- The paper reports RMSE improvements of 9.43% to 20.37% over baselines; the body also states 9.18% over the second-best method, an internal inconsistency.
- Removing adaptive regularization (w/o AR) degrades RMSE on most datasets, e.g., BD 0.2536 vs 0.2397 for APAR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use APAR instead of the strongest listed baseline | APAR · BD, AM, HS, GS, ER, PM, BS, FP | RMSE: 0.2404 -> 0.2397; 0.2352 -> 0.2293; 0.3333 -> 0.3305; 0.1249 -> 0.1205; 0.0422 -> 0.0338; 0.5537 -> 0.5239; 0.0244 -> 0. | Helped |
| Use APAR instead of TabNet* or MLP | APAR · YE, KP | RMSE: 0.2114 vs 0.2148; 0.0500 vs 0.0500 | Mixed |
| Add arithmetic-aware pre-training (ablation) | APAR · tabular regression datasets | RMSE: 0.2520 -> 0.2397; 0.6173 -> 0.5239 | Helped |
| Replace arithmetic-aware pre-training with feature reconstruction (ablation) | APAR · tabular regression datasets | RMSE: 0.2468 vs 0.2397; 0.0297 vs 0.0338 | Mixed |
| Replace arithmetic-aware pre-training with mask reconstruction (ablation) | APAR · tabular regression datasets | RMSE: 0.3582 vs 0.3305; 0.1956 vs 0.1205 | Hurt |
| Replace arithmetic-aware pre-training with feature and mask reconstruction (ablation) | APAR · tabular regression datasets | RMSE: 0.2112 vs 0.2148; 0.2508 vs 0.2397 | Mixed |
| Add adaptive-regularized fine-tuning (ablation) | APAR · tabular regression datasets | RMSE: 0.0632 -> 0.0139; 0.5955 -> 0.5239 | Helped |
| Use multiplication instead of addition for arithmetic-aware pre-training (ablation) | APAR · tabular regression datasets | RMSE: 0.2397 vs 0.2495; 0.3458 vs 0.3305; 0.1205 vs 0.1453 | Mixed |
| Use subtraction instead of addition for arithmetic-aware pre-training (ablation) | APAR · tabular regression datasets | RMSE: 0.2408 vs 0.2495; 0.2413 vs 0.2293; 0.3422 vs 0.3305 | Mixed |
| Use division as the arithmetic-aware pre-training operation (ablation) | APAR · AM, ER, PM, BS, FP | RMSE: - | Hurt |
Surprising findings
- Division-based arithmetic pre-training often failed to converge (missing results on 6 of 10 datasets), and addition/multiplication worked better than subtraction/division.
Limitations
- Arithmetic operation choice must be selected empirically and depends on label scale; division often fails to converge.
- Results averaged over only 5 seeds with small reported gaps on some datasets (e.g., ER tie with w/o AR at 0.0266).
Experimental setup
Datasets: Taiwan Housing (BD, AM, HS) Gas Emission (GS) Election Results (ER) Beijing PM2.5 (PM) Bike Sharing (BS) Year (YE) Kernel Performance (KP) Flight Price (FP)
Benchmarks: Taiwan Housing (BD, AM, HS) Gas Emission (GS) Election Results (ER) Beijing PM2.5 (PM) Bike Sharing (BS) Year (YE) Kernel Performance (KP) Flight Price (FP)
Metrics: RMSE average rank
Baselines: XGBoost LightGBM CatBoost MLP AutoInt FT-Transformer VIME TabNet TabTransformer TabR Mambular
Builds on: FT-Transformer TabNet VIME Gaussian copula feature selection
Closest papers by meaning
- Hybrid Autoencoders for Tabular Data: Leveraging Model-Based Augmentation in Low-Label Settings (NeurIPS 2025 · similarity 0.84)
- TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fields (NeurIPS 2025 · similarity 0.84)
- TabArena: A Living Benchmark for Machine Learning on Tabular Data (NeurIPS 2025 · similarity 0.84)
- PTaRL: Prototype-based Tabular Representation Learning via Space Calibration (ICLR 2024 · similarity 0.84)
- Strengthening LLMs for Tabular Prediction with Structural Priors (NeurIPS 2026 · similarity 0.84)
- TabDPT: Scaling Tabular Foundation Models on Real Data (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.