Research map / Label-efficient and robust learning / Recognition and long-tail
Revisiting Nearest Neighbor for Tabular Data: A Deep Tabular Baseline Two Decades Later
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Revisits NCA (neighbourhood components analysis) for tabular data, progressively adding SGD training, higher-dimensional linear projection, soft-NN inference, deep MLP embeddings with PLR encoding and stochastic neighborhood sampling to yield ModernNCA, a deep tabular baseline competitive with CatBoost.
Problem
Deep tabular models still struggle to match tree-based methods like GBDT on tabular data; classical nearest-neighbor methods such as NCA have been overlooked for tabular prediction.
Approach
Starting from NCA and adding modern components step by step (SGD optimization, unrestricted projection dimension, log-likelihood loss, soft-NN prediction, batch-norm MLP embeddings with PLR numerical encoding, and stochastic neighborhood sampling as mini-batch candidate neighbors) yields a nearest-neighbor model that is competitive with CatBoost and outperforms other deep tabular models.
Key results
- ModernNCA ranks first in classification and is just behind CatBoost in regression across 300 datasets, outperforming other deep tabular models.
- ModernNCA wins on 114 datasets and ties on 81 compared to CatBoost; beats TabR on 123 datasets with 108 ties.
- Ablation on 45 tiny-benchmark datasets: average classification rank improves from NCAv0 4.400 to NCAv4 2.962 (linear L-NCA) via higher dimension, SGD, log loss and soft-NN prediction.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| allow projection to higher dimensions (ablation) | NCA · tiny benchmark, 27 classification datasets | average rank: 4.400 -> 3.708 | Helped |
| replace L-BFGS with SGD (ablation) | NCA · tiny benchmark, 27 classification datasets | average rank: 3.708 -> 3.296 | Helped |
| use log loss instead of the original probability-sum loss (ablation) | NCA · tiny benchmark, 27 classification datasets | average rank: 3.296 -> 3.192 | Helped |
| use soft-NN prediction instead of hard KNN prediction (ablation) | NCA · tiny benchmark, 27 classification datasets | average rank: 3.192 -> 2.962 | Helped |
| replace the nonlinear MLP mapping with a linear mapping (ablation) | ModernNCA · tiny benchmark, classification | average rank: 2.333 -> 2.778 | Hurt |
| replace batch normalization with layer normalization (ablation) | ModernNCA · tiny benchmark, classification | average rank: 2.333 -> 2.537 | Hurt |
| add a residual connection (ablation) | ModernNCA · tiny benchmark, regression | average rank: 2.333 -> 2.806 | Hurt |
| add PLR numerical encoding (ablation) | ModernNCA · tiny benchmark, classification | average rank: 3.037 -> 2.630 | Helped |
| add PLR numerical encoding (ablation) | ModernNCA · tiny benchmark, regression | average rank: 3.389 -> 3.222 | Helped |
| replace Euclidean distance with cosine distance (ablation) | ModernNCA · tiny benchmark, classification | average rank: 2.593 -> 2.111 | Helped |
| replace Euclidean distance with cosine distance (ablation) | ModernNCA · tiny benchmark, regression | average rank: 2.500 -> 2.529 | Hurt |
| use class-wise label-based sampling instead of random sampling (ablation) | ModernNCA · tiny benchmark, classification | average rank: 1.869 -> 2.230 | Hurt |
Surprising findings
- Replacing scikit-learn's L-BFGS optimizer with SGD alone already yields a notable accuracy boost for NCA on tabular tasks.
- Linear NCA with soft-NN prediction surpasses MLP in classification rank on the tiny benchmark.
Limitations
- Does not explicitly model temporal relationships, so it struggles under distribution shift with temporal splits.
- Performance degrades on high-dimensional datasets where d is much larger than N due to curse of dimensionality.
Experimental setup
Datasets: 300-dataset tabular benchmark (UCI, OpenML, Kaggle, others) 45-dataset tiny benchmark
Benchmarks: 300-dataset tabular benchmark 45-dataset tiny benchmark
Metrics: accuracy RMSE average performance rank win/tie/lose with t-test training time memory usage
Baselines: CatBoost XGBoost LightGBM RandomForest SVM KNN TabR MLP MLP-PLR FT-Transformer ResNet SAINT
Builds on: Neighbourhood Components Analysis TabR soft-NN loss PLR embeddings
Closest papers by meaning
- TabR: Tabular Deep Learning Meets Nearest Neighbors (ICLR 2024 · similarity 0.91)
- Retrieval & Fine-Tuning for In-Context Tabular Models (NeurIPS 2024 · similarity 0.88)
- A Data-Centric Perspective on Evaluating Machine Learning Models for Tabular Data (NeurIPS 2024 · similarity 0.87)
- TabM: Advancing tabular deep learning with parameter-efficient ensembling (ICLR 2025 · similarity 0.87)
- TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarks (ICLR 2025 · similarity 0.86)
- Representation Space Augmentation for Effective Self-Supervised Learning on Tabular Data (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.