Research map / Large language models / Language, safety and interpretability
Sample then Identify: A General Framework for Risk Control and Assessment in Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TRON, a two-step split-conformal framework that first calibrates how many responses to sample so an acceptable answer is covered, then uses a frequency-based nonconformity score to filter responses, giving risk control for any sampling-capable MLLM on open-ended VideoQA.
Problem
Split conformal prediction for risk control in MLLMs is limited to logit-based or multiple-choice settings, and open-ended VideoQA lacks fixed options and reliable black-box confidence, so guarantees and risk assessment fail for API-only models.
Approach
A sampling-count conformal score determines how many responses to draw so that an acceptable answer is covered with risk alpha; a nonconformity score of one minus the semantic-cluster frequency of the acceptable response then selects high-quality responses with risk beta, yielding joint miscoverage bound alpha+beta-alpha*beta. Deduplicated prediction set size is proposed as an uncertainty-based ev
Key results
- TRON keeps empirical miscoverage below the bound alpha+beta-alpha*beta (0.19 at alpha=beta=0.1) across four VideoQA datasets and eight MLLMs.
- On VMME with VideoLLaMA-7B, deduplicated APSS is 2.74 under semantic diversity versus 2.81 under frequency, and PandaGPT-7B drops from 2.70 to 2.42.
- On MMLU with LLaMA-3-8B-Instruct, TRON with M=20 reaches APSS 2.76 versus 2.93 for LAC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace frequency with semantic diversity as the reliability measure (ablation) | TRON · VMME | APSS: 2.81 -> 2.74; 2.58 -> 2.52; 2.70 -> 2.42; 2.24 -> 2.20 | Helped |
| Replace frequency with semantic diversity as the reliability measure (ablation) | TRON · VMME | EER: 0.1793 -> 0.1766; 0.1767 -> 0.1744; 0.1812 -> 0.1675; 0.1783 -> 0.1674 | No clear effect |
| Deduplicate semantically redundant responses in prediction sets (ablation) | TRON · Open-ended VideoQA | APSS stability across risk levels | Helped |
| Increase sampling size from M=5 to M=20 (ablation) | TRON with LLaMA-3-8B-Instruct · MMLU | APSS: 3.06 -> 2.76 | Helped |
| Increase sampling size from M=10 to M=20 (ablation) | TRON with LLaMA-3.1-8B-Instruct · MMLU | APSS: 2.53 -> 2.50 | Helped |
| Use semantic diversity instead of frequency for reliability measurement (ablation) | TRON with Gemini-1.5-Pro · VMME | maximum stratified miscoverage rate: 0.2065 -> 0.1667 | Helped |
| Add audio information to video and text input (ablation) | MLLMs evaluated on VideoQA · VideoQA | ACC | Helped |
| Increase Silence Percentage in audio (ablation) | MLLMs evaluated on VideoQA · VideoQA | APSS | Mixed |
Surprising findings
- Higher accuracy does not necessarily mean lower risk: PandaGPT-13B has higher ACC than GPT-4o mini on MSVD but a larger APSS.
- Conditional (size-stratified) miscoverage can exceed the risk level, with max miscoverage 0.2065 under frequency on VMME.
Limitations
- Guarantees are marginal over the test set rather than conditional on individual data points.
- Relies on exchangeability; distribution shift is not handled.
- Semantic equivalence depends on a DeBERTa NLI classifier, and conditional coverage can be violated in practice.
Experimental setup
Datasets: Video-MME NExT-QA MUSIC-AVQA MSVD CoQA TriviaQA MMLU VQA
Benchmarks: Video-MME NExT-QA MUSIC-AVQA MSVD CoQA TriviaQA MMLU VQA
Metrics: empirical error rate accuracy average prediction set size
Baselines: LAC
Builds on: split conformal prediction self-consistency least ambiguous set-valued classifiers
Closest papers by meaning
- Large language model validity via enhanced conformal prediction methods (NeurIPS 2024 · similarity 0.88)
- Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approach (ICML 2025 · similarity 0.88)
- SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.88)
- MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Models (NeurIPS 2024 · similarity 0.88)
- Evaluating language models as risk scores (NeurIPS 2024 · similarity 0.88)
- Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.