Research map / Label-efficient and robust learning / Self-supervised and contrastive
Hybrid Autoencoders for Tabular Data: Leveraging Model-Based Augmentation in Low-Label Settings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TANDEM, a hybrid autoencoder pairing a neural encoder and an oblivious soft decision tree encoder with sample-specific stochastic gating, using the tree only during training as model-based augmentation for low-label tabular classification and regression.
Problem
Deep networks underperform tree models on tabular data under low label budgets, and standard SSL augmentations are hard to design for heterogeneous tabular features.
Approach
Jointly trains a neural encoder and an OSDT encoder with a shared decoder, separate sample-specific stochastic gates, and alignment/latent-cosine losses; only the neural encoder is deployed, so the tree acts as a training-time model-based augmentation.
Key results
- At 400 labels, TANDEM reaches mean accuracy 0.7124 on classification, versus 0.7012 for TabPFN (next best) and mean rank 1.58.
- On regression at 400 labels, TANDEM achieves mean MSE 0.3234 and mean rank 3.38, versus 0.3405 for XGBoost (next best MSE).
- Ablation: removing gating drops classification mean accuracy from 0.7124 to 0.6966; SS-AE alone reaches 0.6815.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TANDEM instead of TabPFN | classification models · Classification datasets, 400 labels | mean accuracy: 0.7012 -> 0.7124 | Helped |
| Use TANDEM instead of CatBoost | regression models · Regression datasets, 400 labels | mean MSE: 0.3318 -> 0.3234 | Helped |
| Add the OSDT encoder to the neural autoencoder (ablation) | TANDEM on classification datasets · Classification datasets, 400 labels | mean accuracy: 0.6815 -> 0.7124 | Helped |
| Add gating to SS-AE (ablation) | TANDEM on classification datasets · Classification datasets, 400 labels | mean accuracy: 0.6815 -> 0.6941 | Helped |
| Add the neural encoder to OSDT AE + Gating (ablation) | TANDEM on classification datasets · Classification datasets, 400 labels | mean accuracy: 0.6600 -> 0.7124 | Helped |
| Add gating (ablation) | TANDEM on classification datasets · Classification datasets, 400 labels | mean accuracy: 0.6966 -> 0.7124 | Helped |
| Add latent similarity and alignment losses (ablation) | TANDEM on classification datasets · Classification datasets, 400 labels | mean accuracy: 0.6971 -> 0.7124 | Helped |
| Add the OSDT encoder to the neural autoencoder (ablation) | TANDEM on regression datasets · Regression datasets, 400 labels | mean MSE: 0.3628 -> 0.3235 | Helped |
| Add gating to SS-AE (ablation) | TANDEM on regression datasets · Regression datasets, 400 labels | mean MSE: 0.3628 -> 0.6551 | Hurt |
| Add the neural encoder to OSDT AE + Gating (ablation) | TANDEM on regression datasets · Regression datasets, 400 labels | mean MSE: 0.9207 -> 0.3235 | Helped |
| Add gating (ablation) | TANDEM on regression datasets · Regression datasets, 400 labels | mean MSE: 0.5397 -> 0.3235 | Helped |
| Add latent similarity and alignment losses (ablation) | TANDEM on regression datasets · Regression datasets, 400 labels | mean MSE: 0.5832 -> 0.3235 | Helped |
Surprising findings
- Neural gating acts as a strong low-pass filter while the tree gating retains more high-frequency content, the reverse of what a simple expectation would suggest for a smoothing-only mask.
- The OSDT-only autoencoder with gating performs worse than the SS-AE baseline on both tasks (classification mean accuracy 0.6600; regression mean MSE 0.9207).
Limitations
- Evaluated only in small-sample low-label regimes (50-1000 labels) with small datasets (up to 2,000 pretraining samples per class).
- Gains are reported on a fixed set of OpenML tabular benchmarks; no evaluation on larger-scale or transformer-based tabular models.
- Some ablation and baseline results differ across tables (e.g., TabNet and TabPFN reporting, TabPFN missing on helena).
Experimental setup
Datasets: OpenML classification datasets (19, from Grinsztajn et al. and Shwartz-Ziv & Armon analyses) OpenML regression datasets (13)
Benchmarks: 19 OpenML classification datasets 13 OpenML regression datasets
Metrics: accuracy MSE mean rank Dolan-More performance profiles statistical significance win counts
Baselines: MLogReg TabNet DeepTLF TabM TabPFN XGBoost CatBoost MLP VIME SCARF SubTab
Builds on: NODE stochastic gates (STG) LSPIN VIME SCARF
Closest papers by meaning
- Representation Space Augmentation for Effective Self-Supervised Learning on Tabular Data (AAAI 2025 · similarity 0.90)
- SwitchTab: Switched Autoencoders Are Effective Tabular Learners (AAAI 2024 · similarity 0.89)
- TabM: Advancing tabular deep learning with parameter-efficient ensembling (ICLR 2025 · similarity 0.89)
- TabGLM: Tabular Graph Language Model for Learning Transferable Representations Through Multi-Modal Consistency Minimization (AAAI 2025 · similarity 0.88)
- TRL-Bench: Standardizing Cross-Paradigm Representation-Level Evaluation of Tabular Encoders (NeurIPS 2026 · similarity 0.88)
- TabR: Tabular Deep Learning Meets Nearest Neighbors (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.