Research map / Vision-language and multimodal / Human and embodied modeling
Conformal Alignment: Knowing When to Trust Foundation Models with Guarantees
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Conformal Alignment, which trains an alignment predictor on reference data and applies conformal p-values with Benjamini-Hochberg to select foundation-model outputs whose alignment is certified with finite-sample FDR control.
Problem
Foundation model outputs in high-stakes settings must be trusted only when aligned with human values, but existing conformal methods either cover single-unit predictions or modify outputs, and do not guarantee the quality of selected outputs for downstream use.
Approach
Adapts conformalized selection to alignment: a classifier predicts alignment scores from features of already-generated outputs, and conformal p-values from a held-out calibration set feed a BH procedure that selects units with controlled false discovery rate, without modifying or regenerating outputs.
Key results
- On TriviaQA with LLaMA-2-13B-chat, FDR never exceeds 0.25 and power reaches 0.50 at target FDR 0.05.
- The Self_Eval thresholding baseline fails to control FDR, while Conformal Alignment strictly controls FDR across target levels 0.05 to 0.95.
- A few hundred reference samples (|D|=500) suffice for powerful selection in both QA and CXR report generation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Conformal Alignment instead of thresholding Self_Eval at 1−α | TriviaQA with OPT-13B and LLaMA-2-13B-chat · TriviaQA | FDR: baseline fails to control FDR; Conformal Alignment controls FDR | Helped |
| Use Conformal Alignment instead of thresholding Self_Eval at 1−α | TriviaQA with OPT-13B and LLaMA-2-13B-chat · TriviaQA | power: baseline is less powerful than our procedure | Helped |
| Use Conformal Alignment instead of thresholding Self_Eval at 1−α | CoQA · CoQA | FDR: baseline fails to control FDR for any given α | Helped |
| Use Conformal Alignment instead of thresholding Self_Eval at 1−α | CoQA · CoQA | power: baseline exhibits lower power | Helped |
| Use LLaMA-2-13B-chat instead of OPT-13B | Conformal Alignment · TriviaQA, α=0.05 | power: close to zero -> 0.50 | Helped |
| Use LLaMA-2-13B-chat instead of OPT-13B | Conformal Alignment · TriviaQA | FDR: 0.5 -> 0.25 | Helped |
| Use Deg(J) or Ecc(J) alone instead of Self_Eval or Num_Sets alone (ablation) | TriviaQA alignment predictor with OPT-13B · TriviaQA | power: least powerful -> most powerful | Helped |
| Use Jaccard-based features rather than NLI-based features alone (ablation) | MIMIC-CXR alignment predictor · MIMIC-CXR | power: NLI-based features exhibit slightly lower power | Helped |
| Increase γ1 from 0.2 to 0.4 or 0.6 (ablation) | Conformal Alignment with logistic regression · TriviaQA | power: power slightly decreases | Hurt |
| Set γ2 to 0.1 or 0.7 rather than a middle value (ablation) | Conformal Alignment with logistic regression · TriviaQA and CoQA | power: slight power loss | Hurt |
| Increase the reference sample size |D| (ablation) | Conformal Alignment · TriviaQA and MIMIC-CXR | power: slight improvement in power | Helped |
Surprising findings
- The more powerful LLaMA-2-13B-chat tends to be more over-confident, so the self-evaluation baseline is more miscalibrated for it than for OPT-13B.
- Shapley analysis suggests NumSets contributes almost nothing to alignment prediction across classifiers.
Limitations
- Alignment scores rely on proxy metrics (ROUGE-L, CheXbert labels) rather than human evaluation, and CheXbert may diverge from human judgment.
- Power is lower on the CXR task, and the guarantee requires exchangeability between reference and test units.
- Only FDR is controlled; the choice of alignment score remains open.
Experimental setup
Datasets: TriviaQA CoQA MIMIC-CXR
Benchmarks: TriviaQA CoQA MIMIC-CXR
Metrics: FDR power
Baselines: Self_Eval thresholding heuristic
Builds on: conformalized selection conformal prediction semantic entropy P(true) self-evaluation
Closest papers by meaning
- Aligner: Efficient Alignment by Learning to Correct (NeurIPS 2024 · similarity 0.86)
- Aligning Model Properties via Conformal Risk Control (NeurIPS 2024 · similarity 0.86)
- S2D-Align: Shallow-to-Deep Auxiliary Learning for Anatomically-Grounded Radiology Report Generation (AAAI 2026 · similarity 0.85)
- Automatic Radiology Reports Generation via Memory Alignment Network (AAAI 2024 · similarity 0.85)
- Learning Causal Alignment for Reliable Disease Diagnosis (ICLR 2025 · similarity 0.85)
- DART: Disease-aware Image-Text Alignment and Self-correcting Re-alignment for Trustworthy Radiology Report Generation (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.