Research map / Label-efficient and robust learning / Self-supervised and contrastive
Representation Space Augmentation for Effective Self-Supervised Learning on Tabular Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RaTab, a representation-space augmentation for contrastive plus reconstruction SSL on tabular data that applies truncated SVD to the last encoder layer's weights combined with dropout.
Problem
Designing effective augmentations for self-supervised learning on tabular data, where input-level augmentations (masking, shuffling, cutmix) can corrupt critical patterns or produce unrealistic samples.
Approach
Shifts augmentation from the raw input space to the representation space by truncating the SVD of the encoder's final-layer weights to create positive views, combined with dropout for diversity and a hybrid InfoNCE plus reconstruction loss.
Key results
- RaTab (on MLP) reaches 0.881 accuracy on CH versus 0.853 for the best input-level/latent baseline (contrastive mix-up) and 0.858-0.869 for GBDTs.
- RaTab (on T2G-former) achieves the top accuracy on most classification datasets, e.g. 0.912 on BM, and lowest MSE on several regression datasets such as CPU (2.489).
- Ablation shows the hybrid loss (InfoNCE plus reconstruction) outperforms either loss alone on all 13 datasets, and applying RaTab to the last layer works best.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RaTab representation-space augmentation instead of masking, shuffling, CutMix, or contrastive mix-up | MLP · CH | accuracy: 0.853 -> 0.881 | Helped |
| Use RaTab representation-space augmentation instead of masking, shuffling, CutMix, or contrastive mix-up | MLP · CS | accuracy: 0.700 -> 0.820 | Helped |
| Use RaTab representation-space augmentation instead of masking, shuffling, CutMix, or contrastive mix-up | MLP · DIA | mean squared error: 570.743 -> 563.975 | Helped |
| Add reconstruction loss to contrastive loss (ablation) | RaTab with MLP · GE | accuracy: 0.581 -> 0.694 | Helped |
| Add contrastive loss to reconstruction loss (ablation) | RaTab with MLP · GE | accuracy: 0.582 -> 0.694 | Helped |
| Add reconstruction loss to contrastive loss (ablation) | RaTab with MLP · DIA | mean squared error: 747.455 -> 527.028 | Helped |
| Add contrastive loss to reconstruction loss (ablation) | RaTab with MLP · FI | mean squared error: 9128.519 -> 8451.140 | Helped |
| Apply RaTab to the third layer rather than the first layer (ablation) | 3-layer MLP · PO | accuracy: 0.898 -> 0.924 | Helped |
| Apply RaTab to the third layer rather than the second layer (ablation) | 3-layer MLP · PO | accuracy: 0.909 -> 0.924 | Helped |
| Apply RaTab to the third layer rather than the second and third layers (ablation) | 3-layer MLP · PO | accuracy: 0.909 -> 0.924 | Helped |
| Apply RaTab to the third layer rather than all layers (ablation) | 3-layer MLP · PO | accuracy: 0.900 -> 0.924 | Helped |
| Use the hybrid reconstruction and contrastive losses rather than contrastive loss alone (ablation) | RaTab with MLP · CS | accuracy: 0.790 -> 0.820 | Helped |
Surprising findings
- Input-level augmentations such as masking and shuffling sometimes underperform a supervised-only baseline, and contrastive mix-up in latent space generally beats them.
- SVD on multiple layers does not further improve performance, attributed to information loss from repeated SVD.
Limitations
- Effectiveness outside tabular data is not validated.
- SVD is applied only to one layer (typically the last) for computational reasons, and multi-layer application did not help.
- Baseline numbers for many DNNs are copied from original papers rather than re-tuned, so comparisons may not be fully controlled.
Experimental setup
Datasets: OpenML tabular datasets (13 datasets: CH, AD, BM, CS, HI, PO, GE, HE, OT, CPU, DIA, FI, HO)
Benchmarks: OpenML 13 tabular datasets (binary, multiclass, regression)
Metrics: accuracy mean squared error
Baselines: XGBoost CatBoost LightGBM MLP FT-Transformer ResNet NODE TabNet DCNv2 SAINT SCARF T2G-former
Builds on: SCARF SAINT Contrastive Mix-up InfoNCE randomized SVD
Closest papers by meaning
- Hybrid Autoencoders for Tabular Data: Leveraging Model-Based Augmentation in Low-Label Settings (NeurIPS 2025 · similarity 0.90)
- PTaRL: Prototype-based Tabular Representation Learning via Space Calibration (ICLR 2024 · similarity 0.90)
- SwitchTab: Switched Autoencoders Are Effective Tabular Learners (AAAI 2024 · similarity 0.89)
- T-JEPA: Augmentation-Free Self-Supervised Learning for Tabular Data (ICLR 2025 · similarity 0.88)
- TabM: Advancing tabular deep learning with parameter-efficient ensembling (ICLR 2025 · similarity 0.87)
- TRL-Bench: Standardizing Cross-Paradigm Representation-Level Evaluation of Tabular Encoders (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.