Research map / Architectures and efficiency / Attention and transformers
InterpreTabNet: Distilling Predictive Signals from Tabular Data by Salient Feature Interpretation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InterpreTabNet, a TabNet variant that models attention masks as Gumbel-Softmax latent variables regularized by a KL-divergence diversity term to yield sparse, interpretable feature masks, plus GPT-4 prompting to narrate them.
Problem
TabNet's attention masks are dense and overlapping across decision steps, making per-instance feature importance hard to interpret; SHAP-based tree models also yield non-sparse importance.
Approach
Treat each TabNet attention mask as a sample from a Gumbel-Softmax latent variable within a cVAE framing, and add a KL-divergence regularizer that pushes adjacent step masks apart, with a search procedure to pick the regularizer weight r_M; masks are then summarized by GPT-4 via few-shot prompts.
Key results
- InterpreTabNet achieves the best accuracy on 2 of 7 datasets (Adult 87.42%, Forest Cover 94.75%) and is competitive on most others.
- In a 20-person blind survey, 65% preferred InterpreTabNet masks for identifying important features (vs. 15% TabNet).
- InterpreTabNet is much worse on Higgs (53.08% vs 72.70% XGBoost) and Mushroom (96.62% vs ~99.9%), and Blastchar/Diabetes also trail.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Original TabNet with InterpreTabNet | TabNet · Adult Census Income | accuracy: 85.55 $\pm$ 0.56 -> 87.42 $\pm$ 0.55 | Helped |
| Replace Original TabNet with InterpreTabNet | TabNet · Forest Cover Type | accuracy: 94.18 $\pm$ 0.63 -> 94.75 $\pm$ 0.53 | Helped |
| Replace TabTransformer with InterpreTabNet | tabular classifier · Poker Hand | accuracy: 99.81 $\pm$ 0.31 -> 99.50 $\pm$ 0.48 | Hurt |
| Replace LightGBM with InterpreTabNet | tabular classifier · Mushroom | accuracy: 100.00 $\pm$ 0.37 -> 96.62 $\pm$ 0.35 | Hurt |
| Replace LightGBM with InterpreTabNet | tabular classifier · Blastchar | accuracy: 77.86 $\pm$ 0.46 -> 72.96 $\pm$ 0.56 | Hurt |
| Replace XGBoost with InterpreTabNet | tabular classifier · Diabetes | accuracy: 61.44 $\pm$ 0.32 -> 55.37 $\pm$ 0.47 | Hurt |
| Replace XGBoost with InterpreTabNet | tabular classifier · Higgs | accuracy: 72.70 $\pm$ 0.35 -> 53.08 $\pm$ 0.56 | Hurt |
| Increase the mask-diversity regularizer r_M (ablation) | InterpreTabNet · Adult Census Income | accuracy | Hurt |
| Increase the mask-diversity regularizer r_M (ablation) | InterpreTabNet · Adult Census Income | feature-mask interpretability | Helped |
| Use GPT-4 to interpret InterpreTabNet feature masks | InterpreTabNet · tabular feature-mask analysis | feature-interdependency interpretation | Helped |
| Present InterpreTabNet rather than the alternatives in the interpretability survey | tabular feature interpretation · Human Evaluation Survey on Interpretability | vote percentage: 65% | Helped |
| Present InterpreTabNet rather than TabNet for GPT-4-generated interpretations | GPT-4 interpretation · Human Evaluation Survey on LLM-Generated Interpretations | vote percentage: 60% | Helped |
Surprising findings
- On Mushroom and Blastchar the sparsity-regularized model's accuracy falls well below baselines at the chosen r_M, showing a notable accuracy cost of sparsity.
- GPT-4 alone without the InterpreTabNet masks could not identify salient features, selecting over 50% of features in one prompt.
Limitations
- Accuracy drops substantially on several datasets (Higgs, Mushroom, Diabetes) relative to tree baselines and original TabNet.
- Human evaluation uses only 20 participants; the authors note larger samples are needed.
- The r_M search is adaptive and dataset-dependent, with very wide ranges (up to 1e13) needed, adding tuning cost.
Experimental setup
Datasets: Adult Census Income Forest Cover Type Poker Hand Mushroom Blastchar Diabetes Higgs
Benchmarks: Adult Census Income Forest Cover Type Poker Hand Mushroom Blastchar Diabetes Higgs
Metrics: test accuracy mask sparsity and salience (qualitative) human preference survey
Baselines: Original TabNet XGBoost LightGBM TabTransformer MLP
Builds on: TabNet Gumbel-Softmax conditional VAE SHAP
Closest papers by meaning
- SwitchTab: Switched Autoencoders Are Effective Tabular Learners (AAAI 2024 · similarity 0.87)
- Interpretable Mesomorphic Networks for Tabular Data (NeurIPS 2024 · similarity 0.86)
- Large Scale Transfer Learning for Tabular Data via Language Modeling (NeurIPS 2024 · similarity 0.86)
- TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fields (NeurIPS 2025 · similarity 0.86)
- Benchmarking Attention for Tabular Foundation Models (NeurIPS 2026 · similarity 0.86)
- Making Pre-trained Language Models Great on Tabular Prediction (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.