Research map / Label-efficient and robust learning / Knowledge distillation
Effective Data Distillation for Tabular Datasets (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a data distillation pipeline for tabular data that encodes one-hot features with a multi-head autoencoder and then applies KMeans clustering in the latent space, outperforming random sampling and KIP across 26 datasets and several classifiers.
Problem
Dataset distillation has been studied for images with neural network target models, but tabular data (one-dimensional, sparse one-hot features, strong non-NN models like XGBoost) poses different challenges and is under-explored.
Approach
Project one-hot tabular data into a low-dimensional latent space with a multi-head autoencoder (reconstruction plus classification loss), then distill with simple clustering (KMeans) centroids rather than gradient-based image methods; find that naive clustering beats KIP for classical models.
Key results
- KMeans in the multi-head MLP autoencoder latent space (KM+MLP⋆) achieves the best average balanced accuracy for most classifiers at N=100, e.g., 0.7279 for MLP and 0.7072 for XGBoost, versus 0.6917 for random sampling on XGBoost.
- Full-data baseline reaches 0.8159 balanced accuracy for XGBoost, so distilled XGBoost still lags the full-data model.
- GNN multi-head encoder uses roughly 4,000 parameters versus about 95,000 for the MLP encoder and performs comparably.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use KMeans in the multi-head MLP autoencoder latent space instead of random sampling | Tabular data distillation · 26 tabular datasets, N=100; XGBoost | balanced accuracy: 0.6917 -> 0.7072 | Helped |
| Use KMeans in the multi-head MLP autoencoder latent space instead of random sampling | Tabular data distillation · 26 tabular datasets, N=100; Logistic Regression | balanced accuracy: 0.6894 -> 0.7267 | Helped |
| Use KMeans in the multi-head MLP autoencoder latent space instead of random sampling | Tabular data distillation · 26 tabular datasets, N=100; MLP classifier | balanced accuracy: 0.6838 -> 0.7279 | Helped |
| Use KMeans in the multi-head MLP autoencoder latent space instead of random sampling | Tabular data distillation · 26 tabular datasets, N=100; NNC | balanced accuracy: 0.6480 -> 0.7265 | Helped |
| Use KMeans in the multi-head MLP autoencoder latent space instead of random sampling | Tabular data distillation · 26 tabular datasets, N=100; NBC | balanced accuracy: 0.6506 -> 0.7195 | Helped |
| Use KMeans in the multi-head GNN autoencoder latent space instead of random sampling | Tabular data distillation · 26 tabular datasets, N=100; XGBoost | balanced accuracy: 0.6917 -> 0.7038 | Helped |
| Use KMeans in the multi-head MLP autoencoder latent space instead of KIP in the same latent space | Tabular data distillation · 26 tabular datasets, N=100; Logistic Regression | balanced accuracy: 0.7098 -> 0.7267 | Helped |
| Use KMeans in the multi-head MLP autoencoder latent space instead of KIP in the same latent space | Tabular data distillation · 26 tabular datasets, N=100; XGBoost | balanced accuracy: 0.6886 -> 0.7072 | Helped |
| Distill with KMeans in multi-head MLP latent space rather than the original representation | Tabular data distillation · 26 tabular datasets, N=100; XGBoost | balanced accuracy: 0.5698 -> 0.7072 | Helped |
| Distill with KMeans in multi-head MLP latent space rather than the original representation | Tabular data distillation · 26 tabular datasets, N=100; Logistic Regression | balanced accuracy: 0.7285 -> 0.7267 | No clear effect |
Surprising findings
- Naive KMeans clustering outperforms the gradient-based image distillation method KIP on tabular data, suggesting gradient-based image methods do not transfer directly.
- The multi-head fine-tuning with classification loss degrades decoder reconstruction, yet the encoder still captures label information in latent space.
Limitations
- Only binary-label datasets considered in experiments
- Full-data XGBoost still substantially outperforms all distilled variants
- Gap to the full-data baseline is large for distilled models at N=100
Experimental setup
Datasets: 26 tabular datasets (10k-110k rows, 7-80 features)
Benchmarks: 26 tabular datasets
Metrics: balanced accuracy
Baselines: random sampling KIP KMeans in one-hot space Agglomerative clustering
Builds on: kernel induced points dataset distillation (Wang et al. 2020) row-column bipartite GNN autoencoder (Wu, Yang, and Yan 2021)
Closest papers by meaning
- Large Scale Dataset Distillation with Domain Shift (ICML 2024 · similarity 0.87)
- What is Dataset Distillation Learning? (ICML 2024 · similarity 0.87)
- Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets? (NeurIPS 2026 · similarity 0.87)
- Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.86)
- DD-Ranking: Rethinking the Evaluation of Dataset Distillation (NeurIPS 2026 · similarity 0.85)
- What Makes a Good Dataset for Knowledge Distillation? (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.