Research map / Vision-language and multimodal / Multimodal LLMs

Proof-of-Perception: Certified Tool-Using Multimodal Reasoning with Compositional Conformal Guarantees

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Proof-of-Perception (PoP), a tool-using multimodal reasoning framework where a DAG of perception/logic nodes emits split-conformal set-valued outputs and a learned controller allocates compute (accept/retry/expand/abort) under a budget.

Problem

Multimodal LLMs on document, chart, and multi-image QA produce single-valued intermediate perception outputs and confident unsupported answers, while tool-use pipelines govern compute heuristically and calibrate at most the final answer.

Approach

Apply split conformal prediction to every node of an executable reasoning graph (OCR, detection, chart parsing, logic fusion) so intermediate outputs are sets with coverage guarantees, and use these set-geometry certificates as the state for a policy-gradient controller that decides when to expand compute.

Key results

• On DocVQA-T1, PoP reaches 78.6 EM vs 75.0 for the best baseline (ProgVLM) and lowers hallucination to 7.1% from 11.9%.
• Ablation on TextVQA: removing conformal sets (No-CP) drops F1 from 72.9 to 69.8 and raises hallucination from 9.3 to 14.8.
• Node-wise coverage reported at 90.2–91.3% against a 90% target across the four node types.

Surprising findings

• Answer-level coverage (88.6%) falls below the 90% target even though per-node guarantees hold, because composition is not covered by the theory.

Limitations

• Guarantees are marginal and per-node; answer-level coverage is not theoretically guaranteed under composition.
• Compute cost model is simple and ignores latency, GPU/CPU overlap and IO.
• Rare corner cases (unseen fonts) may need larger candidate sets to maintain coverage.

Experimental setup

Datasets: DocVQA TextVQA InfographicVQA ChartQA ChartQA-PoT MultiDoc2Dial TextCaps RVL-CDIP-style documents

Benchmarks: DocVQA TextVQA InfographicVQA ChartQA-H MultiDoc2Dial TextCaps

Metrics: EM F1 CIDEr AbsErr hallucination rate empirical coverage average budget

Baselines: Direct-MLLM M-CoT MM-ReAct ProgVLM

Builds on: split conformal prediction ReAct program-of-thought chain-of-thought

Closest papers by meaning

• Perceptual-Evidence Anchored Reinforced Learning for Multimodal Reasoning (CVPR 2026 · similarity 0.88)
• Bad Seeing or Bad Thinking? Rewarding Perception for Multimodal Reasoning (ICML 2026 · similarity 0.86)
• Pause and Reflect: Conformal Aggregation for Chain-of-Thought Reasoning (NeurIPS 2026 · similarity 0.86)
• What MLLMs Learn about When they Learn about Multimodal Reasoning (NeurIPS 2026 · similarity 0.85)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal Reasoning (CVPR 2026 · similarity 0.85)
• See It, Say It, Sorted: An Iterative Training-Free Framework for Visually-Grounded Multimodal Reasoning in LVLMs (CVPR 2026 · similarity 0.84)