Research map / Label-efficient and robust learning / Recognition and long-tail
CACTI: Leveraging Copy Masking and Contextual Information to Improve Tabular Data Imputation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CACTI, a transformer masked autoencoder for tabular imputation that combines median truncated copy masking (using empirical missingness patterns) with column-name/description language-model embeddings, outperforming 13 baselines on 10 datasets.
Problem
Tabular data imputation under MCAR, MAR, and MNAR missingness, where existing methods make restrictive missingness assumptions and underuse feature context/semantics.
Approach
Replace random masking in masked autoencoders with median truncated copy masking, which reuses empirical missingness masks from the data and caps observed-feature counts per batch at the batch median to avoid excess null tokens; additionally inject language-model embeddings of column names/descriptions as context in encoder and decoder.
Key results
- CACTI achieves average relative R^2 improvement over next best method of 13.4% (MNAR), 6.1% (MAR), and 5.3% (MCAR) at 30% missingness.
- Ablation: MT-CM gives a 115% R^2 gain over random masking under MNAR; context adds about 5% on top of MT-CM.
- CACTI reaches R^2 of 0.46/0.47/0.46 (MCAR/MAR/MNAR) at 30% missingness, versus 0.44 for the next best (ReMasker, MCAR).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add semantic context embeddings to the MT-CM model (ablation) | CMAE · 10 datasets, MCAR, MAR, and MNAR, 30% missingness | R²: 0.44 -> 0.45; 0.46 -> 0.47; 0.44 -> 0.46 | Helped |
| Replace random masking with MT-CM (ablation) | Transformer imputation model · Four-dataset ablation, MNAR, 30% missingness | R²: 0.20 -> 0.43 | Helped |
| Add context awareness to random masking (ablation) | RMAE · Four-dataset ablation, MNAR, 30% missingness | R²: 0.20 -> 0.26 | Helped |
| Use MT-CM instead of naive copy masking (ablation) | CMAE · bike and obesity average, MCAR, MAR, and MNAR, 30% missingness | R²: 6.7%, 5.8% and 2.1% higher | Helped |
| Increase naive copy masking rate (ablation) | CMAE with naive copy masking · bike and obesity average, all three missingness scenarios | R² | Hurt |
| Increase MT-CM masking rate toward 90% (ablation) | CMAE with MT-CM · bike and obesity average, all three missingness scenarios | R² | Helped |
| Use loss over masked values instead of observed values alone (ablation) | CACTI · Tabular imputation ablation | imputation performance | Helped |
| Combine observed-value and masked-value reconstruction losses (ablation) | CACTI · Tabular imputation ablation | imputation performance | Helped |
| Increase decoder depth to 12 (ablation) | CACTI · Four datasets, MAR, 30% missingness | R²: 0.47 -> 0.43 | Hurt |
| Increase embedding size from 64 to 512 (ablation) | CACTI · Four datasets, MAR, 30% missingness | R²: 0.46 -> 0.38 | Hurt |
| Use GTE-en-MLM-large rather than other tested context embedding models (ablation) | CACTI · Four datasets, MAR and MNAR, 30% missingness | R² | Mixed |
Surprising findings
- Naive copy masking performance decreases as masking rate increases, whereas MT-CM performance increases, peaking at about 90% masking rate.
- NV-Embed-v2 (7B) consistently underperforms BERT-base (110M) as context embedding model, suggesting semantic content matters more than model size.
Limitations
- Copy masking is an approximation of the true missingness mechanism and does not fully model it.
- Context contribution is not statistically significant under MAR at 30% missingness.
- Several baselines (ReMasker, DiffPuter, notMIWAE, AutoComplete) fail to converge on some datasets at higher missingness rates.
Experimental setup
Datasets: California Housing Magic Gamma Telescope Spam Base Letter Recognition Obesity Bike Default of Credit Card Clients Adult Income Online Shoppers Purchasing Intention Students dropout
Benchmarks: California Housing Magic Gamma Telescope Spam Base Letter Recognition Obesity Bike Default of Credit Card Clients Adult Income Online Shoppers Purchasing Intention Students dropout
Metrics: R^2 RMSE Wasserstein distance
Baselines: ReMasker DiffPuter HyperImpute MissForest notMIWAE Sinkhorn ICE AutoComplete MICE GAIN SoftImpute MIWAE
Builds on: masked autoencoder copy masking (AutoComplete) ReMasker HyperImpute
Closest papers by meaning
- ReMasker: Imputing Tabular Data with Masked Autoencoding (ICLR 2024 · similarity 0.93)
- To Predict or Not to Predict? Proportionally Masked Autoencoders for Tabular Data Imputation (AAAI 2025 · similarity 0.90)
- Learning What Not to Impute: An Uncertainty-Aware Diffusion Framework for Meaningful Missingness (NeurIPS 2026 · similarity 0.88)
- TabDiff: a Mixed-type Diffusion Model for Tabular Data Generation (ICLR 2025 · similarity 0.86)
- TabNAT: A Continuous-Discrete Joint Generative Framework for Tabular Data (ICML 2025 · similarity 0.86)
- Discretizing Continuous Time Series for Imputation with Masked Diffusion Training (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.