Research map / Learning theory and algorithms / Prediction and decision losses
Conformal Generative Modeling with Improved Sample Efficiency through Sequential Greedy Filtering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SCOPE-Gen, a sequential conformal prediction method that prunes i.i.d. samples from a black-box generative model with greedy diversity and quality filters, factorizing admissibility as a Markov chain so each step is calibrated with conformal prediction and needs far fewer admissibility queries than CLM.
Problem
Generative models lack statistical guarantees on their outputs; existing conformal language modeling (CLM) needs many costly admissibility checks during calibration, and multi-dimensional calibration parameters cannot be calibrated with standard conformal prediction.
Approach
Generate an initial set of i.i.d. samples, then sequentially prune it with greedy quality and diversity filters; the admissibility of the final set factorizes as a Markov chain, so each step's threshold is calibrated separately with one-dimensional conformal prediction, using early stopping of admissibility checks.
Key results
- On MIMIC-CXR at alpha=0.3, SCOPE-Gen uses about 4.2 admissibility queries versus 20 for CLM (about 21%).
- Calibration time on MIMIC-CXR is 0.072 s for SCOPE-Gen versus 113.677 s for CLM.
- SCOPE-Gen config. 1 gives the smallest prediction sets on TriviaQA with sum non-conformity (1.202 vs 2.011 for CLM).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace CLM with SCOPE-Gen | TriviaQA · TriviaQA | # Queries: 6.570 ± 0.210 -> 2.910 ± 0.229 | Helped |
| replace CLM with SCOPE-Gen | MIMIC-CXR · MIMIC-CXR | # Queries: 20.000 ± 0.000 -> 4.214 ± 1.011 | Helped |
| replace CLM with SCOPE-Gen | CNN/DM · CNN/DM | # Queries: 20.000 ± 0.000 -> 3.456 ± 0.478 | Helped |
| replace CLM with SCOPE-Gen | Molecular graph extension · Molecules | # Queries: 19.280 ± 0.223 -> 8.606 ± 0.594 | Helped |
| reduce CLM max sample limit to 10 | CLM · MIMIC-CXR | Frac. Reject: 0.240 ± 0.000 -> 0.817 ± 0.000 | Hurt |
| remove both filtering stages (ablation) | SCOPE-Gen · TriviaQA | Set Size: 1.202 ± 0.164 -> 2.050 ± 0.184 | Hurt |
| remove both filtering stages (ablation) | SCOPE-Gen · MIMIC-CXR | # Queries: 4.214 ± 1.011 -> 6.518 ± 0.310 | Hurt |
| apply quality filter before diversity filter (ablation) | SCOPE-Gen · MIMIC-CXR | Set Size: 7.390 ± 1.856 -> 7.848 ± 1.976 | Hurt |
| use only diversity filter (config. 2) (ablation) | SCOPE-Gen · MIMIC-CXR | Set Size: 7.390 ± 1.856 -> 5.839 ± 1.144 | Helped |
Surprising findings
- Reducing CLM's max sample limit to 10 raises rejected calibrations to 81.7% on MIMIC-CXR, far worse than SCOPE-Gen.
- CLM produces smaller prediction sets than SCOPE-Gen under the max non-conformity on some tasks.
Limitations
- Experiments use automated admissibility functions rather than human domain experts.
- Admissibility levels below 0.3 are infeasible for CNN/DM and Molecules because the base models rarely generate admissible outputs within max samples.
- Rejected calibrations (fraction up to about 20%) remain for some tasks.
Experimental setup
Datasets: TriviaQA MIMIC-CXR CNN/DM MOSES
Benchmarks: TriviaQA MIMIC-CXR CNN/DM MOSES molecules
Metrics: # queries time set size fraction rejected admissibility
Baselines: CLM CLM reduced max SCOPE-Gen gen. only SCOPE-Gen flipped
Builds on: conformal language modeling conformal risk control learn-then-test Pareto testing
Closest papers by meaning
- Generative Conformal Prediction with Optimized Coverage Allocation (NeurIPS 2026 · similarity 0.89)
- Conformal Prediction Beyond the Seen: A Missing Mass Perspective for Uncertainty Quantification in Generative Models (NeurIPS 2025 · similarity 0.87)
- Conformal Language Modeling (ICLR 2024 · similarity 0.86)
- Conformal Language Modeling via Posterior Sampling (NeurIPS 2026 · similarity 0.86)
- Conformal Autoregressive Generation: Beam Search with Coverage Guarantees (AAAI 2024 · similarity 0.86)
- Conformal Structured Prediction (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.