Research map / Architectures and efficiency / Attention and transformers
SAND: Smooth imputation of sparse and noisy functional data with Transformer networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds SAND, a self-attention-on-derivatives module stacked on a GeLU transformer, to produce smooth imputations of sparse, noisy functional data, with theory on differentiability and hidden-node count.
Problem
Imputing sparse and noisy irregularly-sampled longitudinal/functional data with transformers yields zigzag, non-smooth curves that fit the noise.
Approach
A SAND module applies attention to approximate the derivative of the transformer's imputed curve, then integrates it via cumulative sum with the initial value, so the output is continuously differentiable by construction and trained end-to-end.
Key results
- On simulated data with n_i=30, SAND reaches MSE 146.5 versus 169.8 for GT1 and 189.9 for PACE, with TV 164.6 versus 218.2 for GT1.
- On UK electricity with n_i=8 to 12, SAND gets MSE 26.7 vs 31.2 for GT and 30.1 for PACE, with TV 20.1 vs 23.2 for GT.
- On downstream AdaFNN prediction for Framingham (n_i=3 to 11), SAND achieves MSE 1.76 vs 2.10 for FACE and 2.23 for PACE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SAND after GT | GeLU transformer · Simulation, ni = 30 | MSE: 169.8(3.2) -> 146.5(2.7) | Helped |
| Add SAND after GT | GeLU transformer · Simulation, ni = 8 to 12 | MSE: 436.7(15) -> 410.9(13) | Helped |
| Add SAND after GT | GeLU transformer · Simulation, ni = 3, 4, 5 | MSE: 798.6(35) -> 758.1(43) | Helped |
| Add SAND after GT | GeLU transformer · UK electricity, ni = 30 | MSE: 10.7(1.8) -> 10.0(1.9) | Helped |
| Add SAND after GT | GeLU transformer · Framingham study, ni = 3 to 11 | MSE*: 0.004(0.0002) -> 0.019(0.001) | Hurt |
| Use parameter L2 penalty (ablation) | GT1 · Simulation | MSE: 169.8(3.2); 436.7(15); 798.6(35) | Mixed |
| Use output-curvature penalty (ablation) | GT2 · Simulation | MSE: 174.8(3.5); 433.8(14); 804.5(39) | Mixed |
| Apply PACE smoothing after GT (ablation) | GT1 · Simulation, ni = 30 | MSE: 169.8(3.2) -> 169.0(3.5) | Helped |
| Apply kernel smoothing after GT (ablation) | GT1 · Simulation | MSE: 169.8(3.2) -> 169.8(3.2); 436.7(15) -> 436.7(15); 798.6(35) -> 796.5(35) | Mixed |
| Apply trend filtering after GT (ablation) | GT1 · Simulation | MSE: 169.8(3.2) -> 160.6(3.2); 436.7(15) -> 421.8(14); 798.6(35) -> 783.4(34) | Helped |
| Apply PACE smoothing after GT (ablation) | GT2 · Simulation | MSE: 174.8(3.5) -> 179.9(3.9); 433.8(14) -> 422.6(13); 804.5(39) -> 788.9(39) | Mixed |
| Apply kernel smoothing after GT (ablation) | GT2 · Simulation | MSE: 174.8(3.5) -> 160.8(3.4); 433.8(14) -> 427.7(13); 804.5(39) -> 804.5(39) | Mixed |
Surprising findings
- Penalizing transformer curvature (GT2) or adding an extra self-attention module (ATT) does not fix the non-smoothness, and ATT increases MSE relative to GT variants.
- On the Framingham data, vanilla GT attains lower imputation MSE than SAND, which the authors attribute to overfitting.
Limitations
- Theoretical guarantees hold on training data and rely on M going to infinity; generalization is not analyzed.
- On the Framingham data, SAND has higher MSE than vanilla GT and larger TV than PACE, FACE and CNP in some cases.
Experimental setup
Datasets: simulated functional data (Fourier basis, 40 basis functions) UK household electricity dataset Framingham Heart Study
Benchmarks: simulated functional data UK electricity Framingham Heart Study
Metrics: MSE total variation (TV) downstream MSE
Baselines: PACE FACE mFPCA MICE CNP GAIN 1DS GT
Builds on: transformer PACE functional PCA kernel smoothing
Closest papers by meaning
- Diffusion Transformers for Imputation: Statistical Efficiency and Uncertainty Quantification (NeurIPS 2025 · similarity 0.87)
- Discretizing Continuous Time Series for Imputation with Masked Diffusion Training (NeurIPS 2026 · similarity 0.85)
- DiffPuter: Empowering Diffusion Models for Missing Data Imputation (ICLR 2025 · similarity 0.85)
- ProCTI: Prototype-Refined Global Conditioning for Diffusion-Based Time Series Imputation (NeurIPS 2026 · similarity 0.85)
- Learning What Not to Impute: An Uncertainty-Aware Diffusion Framework for Meaningful Missingness (NeurIPS 2026 · similarity 0.84)
- Rethinking the Diffusion Models for Missing Data Imputation: A Gradient Flow Perspective (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.