Research map / Vision-language and multimodal / Multimodal LLMs
TLDR: Token-Level Detective Reward Model for Large Vision Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TLDR, a token-level reward model for vision-language models that labels each response token as hallucinated or not, trained on synthetic perturbed VQA answers and dense captions, and shows uses for hallucination evaluation, self-correction, and annotation speedup.
Problem
Standard reward models give one scalar or binary score per whole response, which is uninterpretable and biased toward text, so they cannot localize hallucinations in vision-language model outputs.
Approach
Replace the sequence-level reward with a per-token binary classifier head on a VLM's hidden states, trained with token-level labels from taxonomy-driven synthetic hard negatives; the same training also updates the backbone like a likelihood objective.
Key results
- TLDR on Llama-3.2-11B-Vision reaches 88.2 response-level accuracy versus 86.7 for the naive reward model, and 45.7 mAP(neg) on the synthetic test split.
- On BLINK and IsoBench, a TLDR-tuned PaliGemma at tau=0.25 improves by up to 3.7 points, and TLDR-tuned Llama-3.2-11B improves by up to 12.5 points.
- Human annotators fixing PixelProse captions with TLDR guidance took about 32.8 seconds on average versus 110.7 seconds with a binary reward model (about 3x speedup).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train the reward on every response token rather than only the last token | PaliGemma-3B reward model · Synthetic DOCCI test | response-level accuracy: 81.1 -> 83.1 | Helped |
| Train the reward on every response token rather than only the last token | Llama-3.2-11B-Vision reward model · Synthetic DOCCI test | response-level accuracy: 86.7 -> 88.2 | Helped |
| Tune both the multimodal projection and decoder (ablation) | PaliGemma-3B TLDR · Synthetic DOCCI test | response-level accuracy: 52.5 -> 83.1 | Helped |
| Tune the multimodal projection in addition to the decoder (ablation) | PaliGemma-3B TLDR · Synthetic DOCCI test | response-level accuracy: 79.4 -> 83.1 | Helped |
| Tune the multimodal projection in addition to the decoder (ablation) | PaliGemma-3B TLDR · Synthetic DOCCI test | mAP(neg): 38.2 -> 41.3 | Helped |
| Tune the multimodal projection in addition to the decoder (ablation) | PaliGemma-3B TLDR · Synthetic DOCCI test | token-level accuracy: 98.3 -> 98.6 | No clear effect |
| Apply TLDR-tuned backbone with inference τ=0.25 | PaliGemma-3B · WinoGround dense captions | token-level hallucination rate (%): 4.444 -> 0.172 | Helped |
| Apply TLDR-tuned backbone with inference τ=0.25 | Llama-3.2-11B-Vision · WinoGround dense captions | token-level hallucination rate (%): 0.073 -> 0.066 | Helped |
| Give TLDR token-level guidance instead of naive binary RM guidance | GPT-4V self-correction · WinoGround | wins: 2 -> 12 | Helped |
| Give TLDR guidance rather than binary reward model guidance | Human caption correction · PixelProse | average annotation time (seconds): 110.7 -> 32.8 | Helped |
| Use inference τ=1 instead of the original model | PaliGemma-3B · BLINK Count | score: 69.2 -> 12.5 | Hurt |
| Use inference τ=0.25 instead of the original model | Llama-3.2-11B-Vision · BLINK Count | score: 55.0 -> 67.5 | Helped |
Surprising findings
- Training the reward head with LoRA on the projection and decoder also improved the backbone VLM as a side effect, acting as a token-level likelihood objective.
- Response-level accuracy is worse than the naive RM on spatial relationship (60.2 vs 74.1), which the authors attribute to the product-based aggregation letting any single token veto a response.
Limitations
- Response-level aggregation by product lets single tokens veto responses, hurting spatial relationship accuracy.
- Human evaluation covers only 100 WinoGround samples and the annotation speed study uses three annotators.
- Hallucination rates rely on the TLDR model itself as the detector, with no external ground truth for the evaluated models.
Experimental setup
Datasets: Visual Genome (VG100K) DOCCI WinoGround BLINK IsoBench PixelProse MMMU MEGA-Bench
Benchmarks: WinoGround BLINK IsoBench MMMU MEGA-Bench DOCCI test split
Metrics: token-level accuracy sentence-level accuracy response-level accuracy mAP(neg|pos) hallucination rate win/tie/loss
Baselines: Naive binary reward model GPT-4o prompting Random guess
Builds on: LLaVA PaliGemma LoRA DPO PPO
Closest papers by meaning
- Calibrated Self-Rewarding Vision Language Models (NeurIPS 2024 · similarity 0.91)
- VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models (CVPR 2025 · similarity 0.90)
- Leveraging Unpaired Data for Vision-Language Generative Models via Cycle Consistency (ICLR 2024 · similarity 0.89)
- MM-RLHF: The Next Step Forward in Multimodal LLM Alignment (ICML 2025 · similarity 0.88)
- ARM-Thinker: Reinforcing Multimodal Generative Reward Models with Agentic Tool Use and Visual Reasoning (CVPR 2026 · similarity 0.88)
- TransPrune: Token Transition Pruning for Efficient Large Vision-Language Model (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.