Research map / Vision-language and multimodal / Multimodal LLMs

TRACER: Verifiable Generative Provenance for Multimodal Tool-Using Agents

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TRACER, which has multimodal tool-using MLLMs emit sentence-level provenance records linking claims to tool turns and typed relations, verifies them, and uses them for RL rewards and local tool credit, evaluated on the new TRACE-Bench built from ToolVQA.

Problem

Tool-using multimodal agents expose tool trajectories and final answers but do not specify which tool observation supports each claim, making verification and credit assignment hard (the "provenance gap").

Approach

Generate each answer sentence together with a structured provenance record (tool turn, evidence unit, relation in Quotation/Compression/Inference), verify it with schema, turn alignment, source authenticity and relation checks, and use verified provenance as a hard traceability gate and per-tool-turn local credit in group-relative RL.

Key results

• TRACER-RL (Qwen3-VL-8B) reaches 78.23% answer accuracy and 95.72% summary accuracy, 23.80 points above the strongest closed-source tool-augmented baseline (Gemini-2.5-Pro + tool at 54.43%).
• Compared with tool-SFT (4,949 calls), TRACER-RL uses 3,486 test-set tool calls, a 29.56% reduction.
• Removing the citation-efficiency reward drops provenance precision from 89.24% to 74.20% and raises tool calls to 4,251.

Surprising findings

• Adding tools to strong closed-source models often lowers accuracy (e.g., ChatGPT-4o-latest falls from 38.29% to 34.96% with tools).
• LLaVA-v1.5-7B + tool produced 9,684 calls, all counted as tool errors, reaching 0.01% summary accuracy.

Limitations

• Relies on LLM-as-a-judge (Gemini-2.5-Pro) for filtering and traceability evaluation, which may introduce bias.
• Benchmark built on ToolVQA with seven tools and a single 8B trainable model family.
• Provenance verification shows cited evidence supports claims but does not prove causal internal use.

Experimental setup

Datasets: ToolVQA TRACE-Bench

Benchmarks: TRACE-Bench

Metrics: answer accuracy summary accuracy tool call count tool error count traceability accuracy provenance precision provenance recall provenance F1

Baselines: ChatGPT-4o-latest Claude-3-5-sonnet Gemini-2.5-Pro Qwen2-VL-7B-Instruct LLaVA-v1.5-7B Qwen2-VL-2B-Instruct InternVL3.5-7B MiMo-VL-7B-RL Qwen3-VL-8B-Instruct + tool-SFT

Builds on: ToolVQA TROVE GenProve GRPO

Closest papers by meaning

• MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agents (ICLR 2026 · similarity 0.87)
• CrossCheck-Bench: Diagnosing Compositional Failures in Multimodal Conflict Resolution (AAAI 2026 · similarity 0.87)
• MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models (ICML 2026 · similarity 0.87)
• Don't Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs (CVPR 2026 · similarity 0.86)
• AgentVista: Evaluating Multimodal Agent in Ultra-Challenging Realistic Visual Scenarios (NeurIPS 2026 · similarity 0.86)
• SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memory (NeurIPS 2026 · similarity 0.86)