Research map / Probabilistic and scientific modeling / Molecules and proteins

Model-Free Assessment of Simulator Fidelity via Quantile Curves

NeurIPS 2026 · Probabilistic and scientific modeling › Molecules and proteins · approach: Prediction-powered and conformal inference · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a model-free, finite-sample method that builds per-scenario confidence sets for latent real-world parameters, forms a pseudo sim-to-real discrepancy, and estimates its quantile curve to profile LLM simulator fidelity on WorldValueBench.

Problem

Quantifying the distributional sim-to-real gap of a black-box simulator (e.g., LLM survey responders) across scenarios when real and simulated population parameters are only observed through finite, heterogeneous samples; standard predictive/conformal methods target observable outcomes rather than l

Approach

Replace the unobservable per-scenario discrepancy with a pseudo-discrepancy given by the supremum of the loss over a per-scenario confidence set for the real parameter, then take empirical quantiles across scenarios with an index adjustment (affine in tau with slope the average coverage) and adaptive per-scenario coverage gamma_j = g(n_j), yielding finite-sample quantile guarantees.

Key results

• GPT-4o yields the lowest calibrated discrepancy quantile curve across quantile levels, with GPT-5 mini close behind and Llama 3.3 and Qwen-3 trailing; all LLMs beat the uniform baseline on over 80% of questions but fail in tail scenarios.
• With adaptive gamma_j = 1 - n_j^(-1/3) (average coverage about 0.87), the calibrated envelope is conservative at n_j=50 but tightens for n_j >= 500 relative to the oracle curve.
• With fixed gamma_j = 1/2, a persistent vertical gap to the oracle curve remains even for large n_j, motivating scenario-dependent coverage.

Surprising findings

• Several pre-trained LLMs underperform a baseline that randomly picks answers in the survey setting.
• With fixed gamma = 1/2, the calibrated curve stays vertically conservative even as human sample size grows, because the index adjustment pushes lower quantiles toward the median.

Limitations

• Conservative envelope when number of scenarios m or per-scenario sample sizes are small, due to DKW and worst-case pseudo-discrepancies.
• Targets scenario-level marginal distributions; does not model respondent-level joint dependence across questions.
• Guarantees assume i.i.d. scenarios and static simulators; no temporal dependence or distribution shift.

Experimental setup

Datasets: WorldValueBench World Values Survey

Benchmarks: WorldValueBench

Metrics: calibrated discrepancy quantile curve CVaR AUC empirical coverage vs oracle quantile

Baselines: GPT-4o GPT-5 mini Llama 3.3 70B Qwen 3 235B uniform random baseline

Builds on: Dvoretzky-Kiefer-Wolfowitz inequality Chernoff-Hoeffding bound for multinomial conformal prediction input uncertainty quantification

Closest papers by meaning

• Addressing Misspecification in Simulation-based Inference through Data-driven Calibration (ICML 2025 · similarity 0.87)
• G-Sim: Generative Simulations with Large Language Models and Gradient-Free Calibration (ICML 2025 · similarity 0.87)
• Social Choice Foundations for Simulation-Augmented Generation (NeurIPS 2026 · similarity 0.85)
• Simulation-Based Inference with Quantile Regression (ICML 2024 · similarity 0.85)
• Consistency Models for Scalable and Fast Simulation-Based Inference (NeurIPS 2024 · similarity 0.85)
• Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors (NeurIPS 2026 · similarity 0.85)