Research map / Label-efficient and robust learning / Self-supervised and contrastive
T-JEPA: Augmentation-Free Self-Supervised Learning for Tabular Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes T-JEPA, an augmentation-free SSL method that pretrains tabular transformers by predicting latent representations of one feature subset from another, and shows gains for MLP, ResNet, and other deep models over raw data and prior SSL methods.
Problem
Self-supervised learning for tabular data typically relies on data augmentations that are hard to design for heterogeneous tabular features, and deep models still lag behind gradient-boosted trees on many tabular tasks.
Approach
Adapts JEPA to tabular data by masking feature subsets and predicting, in latent space via an EMA target encoder and a predictor, the representation of a non-overlapping feature subset from the context subset; also adds regularization [REG] tokens to avoid initial representation collapse.
Key results
- ResNet+T-JEPA achieves the lowest average rank (2.6) among SSL-augmented MLP/ResNet variants, versus 10.4 for raw ResNet.
- T-JEPA improves MLP accuracy on Adult from 0.827 to 0.866 and on Jannis from 0.672 to 0.728.
- Augmented deep models often match or exceed XGBoost and CatBoost; ResNet+T-JEPA outperforms GBDT on most datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add T-JEPA pre-training to MLP | MLP · Adult, Helena, Jannis, ALOI, California housing, Higgs, and MNIST | accuracy / RMSE: Adult accuracy: 0.827 -> 0.866; Helena accuracy: 0.353 -> 0.400; Jannis accuracy: 0.672 -> 0.728; ALOI accuracy: 0.916 - | Mixed |
| add T-JEPA pre-training to ResNet | ResNet · Adult, Helena, Jannis, ALOI, California housing, Higgs, and MNIST | accuracy / RMSE: Adult accuracy: 0.814 -> 0.865; Helena accuracy: 0.351 -> 0.401; Jannis accuracy: 0.666 -> 0.718; ALOI accuracy: 0.919 - | Helped |
| remove the regularization token [REG] (ablation) | T-JEPA · Jannis | representation collapse | Hurt |
| add one or more regularization tokens [REG] (ablation) | T-JEPA · Jannis | representation collapse | Helped |
| replace dropping masked features with zero-replacement plus mask indicators (ablation) | T-JEPA · tabular data | representation collapse | Hurt |
| replace the Transformer predictor with feature-specific MLP predictors (ablation) | T-JEPA · tabular data | representation collapse | Hurt |
Surprising findings
- Without a regularization token, T-JEPA fails to escape the initial representation collapse on Jannis.
- Attention-based models (FT-Transformer, AutoInt) gained from T-JEPA but did not surpass ResNet and MLP with T-JEPA.
Limitations
- Representations are best suited to transformer-like architectures and need adaptation for other architectures.
- Initial representation collapse requires regularization tokens; the theoretical basis for non-contrastive collapse avoidance is not established.
Experimental setup
Datasets: Adult Higgs Helena Jannis ALOI California housing MNIST
Benchmarks: Adult Higgs Helena Jannis ALOI California housing MNIST
Metrics: accuracy RMSE average rank KL divergence uniformity Kendall's tau
Baselines: MLP DCNv2 ResNet AutoInt FT-Transformer XGBoost CatBoost PTaRL SwitchTab VIME BinRecon SubTab
Builds on: I-JEPA VIME SubTab register tokens ResNet
Closest papers by meaning
- Representation Space Augmentation for Effective Self-Supervised Learning on Tabular Data (AAAI 2025 · similarity 0.88)
- Hybrid Autoencoders for Tabular Data: Leveraging Model-Based Augmentation in Low-Label Settings (NeurIPS 2025 · similarity 0.87)
- How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networks (NeurIPS 2024 · similarity 0.86)
- You Don’t Need Domain-Specific Data Augmentations When Scaling Self-Supervised Learning (NeurIPS 2024 · similarity 0.86)
- SwitchTab: Switched Autoencoders Are Effective Tabular Learners (AAAI 2024 · similarity 0.86)
- Making Pre-trained Language Models Great on Tabular Prediction (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.