Research map / Label-efficient and robust learning / Knowledge distillation

FEAT-KD: Learning Concise Representations for Single and Multi-Target Regression via TabNet Knowledge Distillation

ICML 2025 · Label-efficient and robust learning › Knowledge distillation · approach: Patient consistent distillation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Distills a trained TabNet into a linear combination of short symbolic features (via exhaustive-search DistilSR per TabNet step) to get an interpretable FEAT-style regressor for single- and multi-target regression.

Problem

Interpretable regression: FEAT's genetic-programming search for concise symbolic features is inefficient, while deep tabular models like TabNet are black boxes; the paper asks how to get accurate, compact symbolic models for single and multi-target regression.

Approach

Instead of discovering symbolic features with genetic programming, decompose a trained TabNet into per-step transformed features, restrict each to the top-3 masked input features, distill each into a short symbolic expression with exhaustive-search DistilSR using affine-invariant MSE, then refit a linear regressor over the resulting features; shared features across targets enable multi-target regr

Key results

• On single-target regression, FEAT-KD has the best R2 among symbolic models on most datasets (e.g., Syn2 0.953 vs FEAT 0.936) but TabNet remains higher (e.g., Syn2 0.996).
• FEAT-KD yields statistically significantly smaller models than FEAT variants in 40 of 45 comparisons; model size ~49 versus 57-104 for FEAT variants and ~5.5k-7.2k for TabNet.
• On multi-target benchmarks FEAT-KD beats TabNet on 32 of 54 targets (e.g., enb 0.85 vs 0.434 for target 1).

Measured effects reported in the paper

Surprising findings

• On multi-target benchmarks FEAT-KD outperforms its TabNet teacher on 32 of 54 targets, suggesting a regularization effect.
• Genetic programming in FEAT was found to be unstable, producing models of widely varying sizes across seeds.

Limitations

• TabNet remains more accurate than FEAT-KD on most single-target datasets.
• Multi-target FEAT baseline not available; compared only against TabNet on multi-target tasks.
• Interpretability is measured only by proxy (model size), not by human study.

Experimental setup

Datasets: Syn1-Syn6 (Chen et al., 2018) Rossmann Store Sales bodyfat cpu act 197 cpu act 573 cpu small house 8L houses pm10 puma8NH SARCOS atp1d

Benchmarks: Syn1-Syn6 Rossmann Store Sales PMLB regression datasets (bodyfat, cpu act, cpu small, house 8L, houses, pm10, puma8NH) SARCOS atp1d enb oes 10 rf1 scm1d

Metrics: R2 score model size (number of functions and terminals) Wilcoxon signed-rank p-values (Bonferroni-adjusted)

Baselines: FEAT FEAT-Corr FEAT-CN TabNet

Builds on: FEAT TabNet DistilSR knowledge distillation

Closest papers by meaning

• High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Laws (ICLR 2025 · similarity 0.83)
• TabPrep: Closing the Feature Engineering Gap in Tabular Benchmarks (NeurIPS 2026 · similarity 0.82)
• Explainable Neural Networks with Guarantee: A Sparse Estimation Approach (AAAI 2025 · similarity 0.82)
• InterpreTabNet: Distilling Predictive Signals from Tabular Data by Salient Feature Interpretation (ICML 2024 · similarity 0.81)
• Provable and Efficient Dataset Distillation for Kernel Ridge Regression (NeurIPS 2024 · similarity 0.81)
• DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024 · similarity 0.81)