Research map / Label-efficient and robust learning / Recognition and long-tail
TabR: Tabular Deep Learning Meets Nearest Neighbors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TabR, a feed-forward tabular network with a single kNN-like retrieval module (L2-based key-only similarity, label embeddings and a correction value module), which beats prior retrieval-based and MLP models and outperforms GBDT on a middle-scale benchmark.
Problem
Tabular deep learning has lagged behind gradient-boosted decision trees, and prior retrieval-augmented tabular DL models give only marginal gains over MLPs while being costly.
Approach
A single-head retrieval module in the spirit of kNN: key-only L2 similarity between target and candidate representations (no queries, no scaling), label embeddings added to values, and a learned correction term T(k - k_i) that translates key differences into label-embedding corrections, with context freezing for faster training.
Key results
- TabR achieves the best average rank among DL models (1.3 ± 0.6) on the default benchmark, improving over MLP-PLR (2.0 ± 1.0).
- Tuned TabR ensembles achieve average rank 1.3 vs. 2.4-2.5 for tuned XGBoost, CatBoost and LightGBM on the default benchmark.
- On the middle-scale benchmark, TabR wins against tuned XGBoost in 26 tasks, losing in only 3, and context freeze gives ~7x speedup on full Weather prediction (18h9min to 3h15min).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TabR instead of MLP-PLR | tabular DL models · WE | RMSE: 1.860 -> 1.690 | Helped |
| use TabR instead of MLP-PLR | tabular DL models · CA | RMSE: 0.476 -> 0.400 | Helped |
| use TabR instead of MLP-PLR | tabular DL models · HO | RMSE: 3.056 -> 3.105 | Hurt |
| add context labels to vanilla attention values (ablation) | Step-0 vanilla attention · CA | RMSE: 0.484 -> 0.489 | Hurt |
| replace query-key dot product with key-only L2 similarity (ablation) | Step-1 context labels · CA | RMSE: 0.489 -> 0.418 | Helped |
| add target-dependent key-difference correction to values (ablation) | Step-2 new similarity module · CA | RMSE: 0.418 -> 0.408 | Helped |
| omit similarity scaling and exclude the target from its own context (ablation) | Step-3 new value module · TabR | RMSE: 0.408 -> 0.403 | Helped |
| add context-label embedding (ablation) | Step-2 retrieval module · CA | RMSE: 0.442 -> 0.418 | Helped |
| use L2 similarity instead of dot product (ablation) | Step-2 retrieval module · CA | RMSE: 0.495 -> 0.418 | Helped |
| share query and key weights (ablation) | Step-2 retrieval module · CA | RMSE: 0.488 -> 0.418 | Helped |
| freeze retrieval contexts after 4 epochs (ablation) | TabR-S · CA | RMSE: 0.406 -> 0.409 | No clear effect |
| remove the correction projection on the label embedding space (ablation) | trained TabR-S · CA | RMSE: 0.403 -> 0.465 | Hurt |
Surprising findings
- Adding context labels alone (Step-1) gives no improvement over vanilla attention; gains appear only when combined with key-only L2 similarity.
- Augmenting XGBoost with a nearest-neighbor average-feature retrieval component does not improve it.
Limitations
- Retrieval-based model is slower to train than simple retrieval-free models and GBDT, and does not scale easily to tens or hundreds of millions of points.
- Retrieval is not universally beneficial across datasets; classification value module correction gave no interpretable gains.
- Discovered a data leak in the Facebook comments dataset split inherited from prior work, which inflated results on that dataset.
Experimental setup
Datasets: California housing Higgs Covertype Microsoft Weather prediction Facebook comments volume Yandex MI/CH/CA/HO/AD/DI/OT/HI/BL/WE/CO/MI default benchmark middle-scale GBDT-friendly benchmark (43 tasks, <=50K objects)
Benchmarks: default tabular benchmark (CH, CA, HO, AD, DI, OT, HI, BL, WE, CO, MI) middle-scale benchmark of 43 tasks from Grinsztajn et al.
Metrics: accuracy RMSE average rank win/tie/loss counts training time
Baselines: kNN DNNR DKL ANP SAINT NPT MLP MLP-PLR XGBoost CatBoost LightGBM
Builds on: k-nearest neighbors DNNR NPT MLP-PLR
Closest papers by meaning
- Revisiting Nearest Neighbor for Tabular Data: A Deep Tabular Baseline Two Decades Later (ICLR 2025 · similarity 0.91)
- TabM: Advancing tabular deep learning with parameter-efficient ensembling (ICLR 2025 · similarity 0.91)
- Retrieval & Fine-Tuning for In-Context Tabular Models (NeurIPS 2024 · similarity 0.90)
- TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fields (NeurIPS 2025 · similarity 0.89)
- TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarks (ICLR 2025 · similarity 0.89)
- Large Scale Transfer Learning for Tabular Data via Language Modeling (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.