Research map / Large language models / Language, safety and interpretability
SAFER: Risk-Constrained Sample-then-Filter in Large Language Models
Qingni Wang, Yue Fan, Xin Eric Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add abstention-aware sampling budget calibration (Clopper-Pearson) instead of TRON's calibration | SAFER sampling stage · TriviaQA and CoQA, five LLMs | test-time EER: TRON EER exceeding 0.06 at alpha=0.03 vs SAFER below alpha | Improved |
| add conformalized filtering with risk level beta (ablation) | SAFER prediction sets · TriviaQA, LLaMA-3.1-8B-Instruct | average prediction set size: 7.9 -> 5.5 | Improved |
| increase beta in the filtering stage (ablation) | SAFER · TriviaQA and CoQA | prediction set size and EER: EER stays below alpha+beta-alpha*beta bound | Mixed |
| use Rouge-L instead of sentence similarity as the correctness criterion (ablation) | SAFER · TriviaQA and CoQA | test-time EER | No clear effect |
| reduce calibration-to-test split ratio from 0.5 to 0.1 (ablation) | SAFER · TriviaQA, sentence similarity | test-time EER: 0.5 -> 0.1 | No clear effect |
| filter unreliable answers with entropy-based threshold | SAFER · TriviaQA and CoQA | empirical coverage | Mixed |
Closest papers by meaning
- Large language model validity via enhanced conformal prediction methods (NeurIPS 2024 · similarity 0.91)
- Sample then Identify: A General Framework for Risk Control and Assessment in Multimodal Large Language Models (ICLR 2025 · similarity 0.91)
- Prune 'n Predict: Optimizing LLM Decision-making with Conformal Prediction (ICML 2025 · similarity 0.91)
- Conformal Language Modeling (ICLR 2024 · similarity 0.89)
- Evaluating language models as risk scores (NeurIPS 2024 · similarity 0.89)
- A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
