Research map / Probabilistic and scientific modeling / Variational and generative sampling
Masked Language Modeling Becomes Conditional Density Estimation for Tabular Data Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MaCoDE, a masked-language-model-style transformer that estimates histogram-based conditional densities for mixed-type tabular data, enabling synthetic data generation and multiple imputation.
Problem
Generating synthetic mixed-type tabular data with high machine learning utility requires accurate conditional distribution estimation, and existing MLM-based tabular synthesizers lack distributional guarantees; missing data handling and multiple imputation are also needed.
Approach
Redefine the consecutive multi-class classification in masked language modeling as histogram-based non-parametric conditional density estimation over CDF-transformed continuous columns, with a theoretical bound showing that orderless classification loss minimizes a total variation bound between conditional distributions.
Key results
- On Q1, MaCoDE achieves the best MMD (.007), Wasserstein distance (1.630), SMAPE (.158), F1 (.635), and feature selection (.925) among compared synthesizers, with TabMT and TabDDPM competitive on some metrics.
- MaCoDE with 30% missingness in training data retains machine learning utility close to the complete-data version (e.g., SMAPE .164-.169 vs .158).
- For multiple imputation under MAR at 0.3 missingness, MaCoDE achieves coverage .963 with bias .006, competitive with MIWAE, not-MIWAE and EGC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MaCoDE instead of TabMT | Synthetic tabular data generation · 10 tabular datasets | MMD: .012_{\pm .001} -> .007_{\pm .001} | Helped |
| Use MaCoDE instead of TabMT | Synthetic tabular data generation · 10 tabular datasets | F1: .622_{\pm .024} -> .635_{\pm .023} | Helped |
| Use MaCoDE instead of TabMT | Synthetic tabular data generation · 10 tabular datasets | Model: .528_{\pm .039} -> .599_{\pm .035} | Helped |
| Use MaCoDE instead of TabDDPM | Synthetic tabular data generation · 10 tabular datasets | Feature: .770_{\pm .027} -> .925_{\pm .007} | Helped |
| Train MaCoDE with incomplete data under MCAR, MAR, MNARL, or MNARQ | MaCoDE · 10 tabular datasets, missingness rate 0.3 | SMAPE | Mixed |
| Train MaCoDE with incomplete data under MCAR, MAR, MNARL, or MNARQ | MaCoDE · 10 tabular datasets, missingness rate 0.3 | F1 | Mixed |
| Use MaCoDE(MAR) instead of MICE | Multiple imputation · 5 datasets, MAR missingness rate 0.3 | Bias: .010_{\pm .001} -> .006_{\pm .000} | No clear effect |
| Use MaCoDE(MAR) instead of EGC | Multiple imputation · 5 datasets, MAR missingness rate 0.3 | Coverage: ${.996}_{\pm .004}$ -> $.963_{\pm .009}$ | Hurt |
Surprising findings
- MaCoDE has the worst KL divergence and GoF among top methods (KL .034 vs TabMT .011), indicating marginal fidelity is not its strength.
Limitations
- Theoretical guarantee holds only under Lipschitz continuity of the conditional densities (Assumption 2).
- Performance declines at high missingness rates (F1 drops beyond 0.5 missingness on kings).
- Marginal statistical fidelity (KL, GoF) is weaker than TabMT and the baseline in some metrics.
Experimental setup
Datasets: abalone banknote breast concrete covtype kings letter loan redwine whitewine
Benchmarks: abalone banknote breast concrete covtype kings letter loan redwine whitewine
Metrics: KL divergence GoF test MMD Wasserstein distance SMAPE F1 model selection Spearman correlation feature selection Spearman correlation bias coverage
Baselines: CTGAN TVAE CTAB-GAN CTAB-GAN+ DistVAE TabDDPM TabMT MICE GAIN missMDA VAEAC MIWAE
Builds on: BERT masked language modeling TabMT TabPFN Rubin multiple imputation
Closest papers by meaning
- TabNAT: A Continuous-Discrete Joint Generative Framework for Tabular Data (ICML 2025 · similarity 0.86)
- Synthesizing Privacy-Preserving Text Data via Finetuning *without* Finetuning Billion-Scale LLMs (ICML 2025 · similarity 0.86)
- Scaling up Masked Diffusion Models on Text (ICLR 2025 · similarity 0.86)
- Breaking the Quality–Privacy Tradeoff in Tabular Data Generation via In-Context Learning (NeurIPS 2026 · similarity 0.86)
- PrE-Text: Training Language Models on Private Federated Data in the Age of LLMs (ICML 2024 · similarity 0.85)
- HARMONIC: Harnessing LLMs for Tabular Data Synthesis and Privacy Protection (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.