Research map / Image generation / Autoregressive and latent generation

Sculpting Features from Noise: Reward-Guided Hierarchical Diffusion for Task-Optimal Feature Transformation

NeurIPS 2025 · Image generation › Autoregressive and latent generation · approach: Latent diffusion with learned tokenizers · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Reframes feature transformation as reward-guided latent diffusion: a VAE with a semi-autoregressive decoder encodes feature token sequences, and an LDM sampled with evaluator gradients generates task-optimal feature sets.

Problem

Feature transformation (FT) suffers from huge combinatorial discrete search spaces and from continuous gradient search that is sensitive to initialization and gets stuck in local optima; autoregressive decoding of feature sets is also slow.

Approach

Model FT as generation: a VAE learns a latent space of feature sets with a hierarchical decoder (predict feature count, then autoregress each feature in parallel), and a latent diffusion model is steered at sampling time by gradients of a performance evaluator (reward guidance) to produce high-reward embeddings.

Key results

• DIFFT achieves the best score on all 14 datasets in the overall comparison, e.g., 0.971 F1 on Ionosphere versus 0.942 for the second-best ELLM, and 0.632 on Openml_618 versus 0.477 for MOAT.
• Per-epoch training on Ap-omentum-ovary (10,936 features) takes 19.64 s for DIFFT versus over 500 s for MOAT; inference is at most 0.43 s versus up to 2.33 s for MOAT.
• Ablations on svmguide3 and openml_586 show the full model (0.866 and 0.647) beats NAR, AR, NoR, and CS variants.

Measured effects reported in the paper

Surprising findings

• Removing reward guidance (NoR) still outperforms the continuous-search variant CS, suggesting the latent space and initialization matter more than guidance alone.

Limitations

• Task-specific design requires regenerating feature sets for each downstream model, adding cost when many learners are involved.
• Validated only on tables up to about 10K raw features; scaling to ultra-wide data strains VAE and diffusion model.
• Evaluation uses a single random-forest downstream model in main experiments and five models only on SpectF.

Experimental setup

Datasets: SpectF SVMGuide3 German Credit Messidor Features SpamBase Ap-omentum-ovary Ionosphere Openml_586 Openml_589 Openml_607 Openml_616 Openml_618

Benchmarks: SpectF SVMGuide3 German Credit Messidor Features SpamBase Ap-omentum-ovary Ionosphere Openml_586 Openml_589 Openml_607 Openml_616 Openml_618

Metrics: F1 score 1-relative absolute error training time inference time

Baselines: RDG PCA LDA ERG AFAT NFS TTG GRFG MOAT ELLM

Builds on: MOAT RCGDM DDIM Min-SNR weighting GRFG

Closest papers by meaning

• LaVin-DiT: Large Vision Diffusion Transformer (CVPR 2025 · similarity 0.86)
• Why DiT Models Underperform as Representation Learners without Long Skip Connections (NeurIPS 2026 · similarity 0.86)
• Diff-Plugin: Revitalizing Details for Diffusion-based Low-level Tasks (CVPR 2024 · similarity 0.86)
• Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (CVPR 2025 · similarity 0.86)
• Diff-MoE: Diffusion Transformer with Time-Aware and Space-Adaptive Experts (ICML 2025 · similarity 0.86)
• Masked Autoencoders Are Effective Tokenizers for Diffusion Models (ICML 2025 · similarity 0.86)