Research map / Large language models / Language, safety and interpretability

Guided Data Generation for Understanding Model Behavior

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Data and evaluation methodology · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a Gibbs-distribution framework that guides MCMC (MALA) sampling of inputs with a loss built from trained models, to generate data answering behavioral questions such as model disagreement, prediction risk, parameter sensitivity, and fixed-label generation.

Problem

Understanding how trained models behave beyond standard predictive accuracy by finding which inputs would make a model exhibit a specified behavior, where static test sets rarely reveal such regions.

Approach

Encode a behavioral question as a guidance function G built from trained model(s) (integrated over a parameter distribution q), and sample from the Gibbs distribution p*(x) ∝ exp(-G(x)/τ) with MALA (optionally in a VAE latent space) to produce inspection datasets; analytic Gaussian solution for linear regression.

Key results

• On FICO, generated samples make XGBoost and logistic regression disagree on 90.1% of samples, versus 5.5% disagreement on the test set.
• Prediction-risky FICO samples from an MLP have mean predicted Bad probability 0.525 (SD 0.017) over 500 samples.
• Goldfish-guided ResNet50 generation yields orange-dominant images; swapping green/blue channels barely hurts goldfish detection, suggesting reliance on red channel.

Surprising findings

• Generated samples show model disagreement on 90.1% of inputs even though the two models agree on 94.5% of test observations.
• Swapping green and blue channels does not substantially impair goldfish detection, suggesting the model relies mainly on the red channel.

Limitations

• Generated samples may not be realistic or respect feature constraints; authors note implicit feature constraints are not enforced.
• Evaluation is on small-scale tabular and MNIST/ResNet50 demos; no quantitative validation against downstream utility.
• Tree-based models have locally flat gradients requiring smoothed-gradient approximations.

Experimental setup

Datasets: FICO (HELOC) Housing MNIST Wine Recognition

Benchmarks: FICO (HELOC) MNIST Housing Wine Recognition

Metrics: disagreement rate accuracy kernelized Stein discrepancy effective sample size Gelman-Rubin R-hat mean predicted probability

Baselines: XGBoost logistic regression Support Vector Regression LeNet5

Builds on: energy-based models Langevin dynamics Metropolis-adjusted Langevin algorithm Kernelized Stein discrepancy variational VAE decoder

Closest papers by meaning

• Generative Data Mining with Longtail-Guided Diffusion (ICML 2025 · similarity 0.83)
• Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferences (NeurIPS 2024 · similarity 0.83)
• Performative Prediction with Selective Labels (NeurIPS 2026 · similarity 0.82)
• Domain Guidance: A Simple Transfer Approach for a Pre-trained Diffusion Model (ICLR 2025 · similarity 0.82)
• Position: Data-driven Discovery with Large Generative Models (ICML 2024 · similarity 0.82)
• Data-Driven Discovery of Design Specifications (Student Abstract) (AAAI 2024 · similarity 0.82)