AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

Reflective Instruction Tuning: Mitigating Hallucinations in Large Vision-Language Models

Jinrui Zhang, Teng Wang, Haigang Zhang, Ping Lu, Feng Zheng

ECCV 2024 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning · 28 citations (OpenAlex)

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add reflective rationale learning with REVERIE to LLaVA-1.0-7b-loraLLaVA-1.0-7b-lora · POPEscore: 71.1 -> 83.8Improved
add REVERIE to LLaVA-1.0-7b-loraLLaVA-1.0-7b-lora · MME / MMBenchscore: MME 819.8 -> 1168.1; MMBench 27.2 -> 55.4Improved
add REVERIE to LLaVA-1.5-7b-lora (vs adding only ScienceQA data)LLaVA-1.5-7b-lora · ScienceQA-IMGaccuracy: 76.3 -> 80.5Improved
add REVERIE to LLaVA-1.5-7b-loraLLaVA-1.5-7b-lora · POPE / MMBench / MM-Vetscore: POPE 86.6 -> 86.4; MMBench 67.9 -> 67.3No clear effect
add only ScienceQA response data (no rationales) to LLaVA-1.5 training (ablation)LLaVA-1.5-7b-lora · MME / GQAscore: MME 1471.0 -> 1439.2; GQA 62.8 -> 60.7Hurt
add positive and negative rationales together versus response only (ablation)REVERIE-1.0-7b-lora · POPE AdversarialF1: 78.49 -> 80.31Improved
add only positive rationales (ablation)REVERIE-1.0-7b-lora · POPE RandomAccuracy: 86.87 -> 85.60Hurt
add both rationales versus response only (ablation)REVERIE-1.0-7b-lora · MMBenchaccuracy: 51.70 -> 55.44Improved
add negative rationales on top of positive rationales (ablation)REVERIE-1.0-7b-lora · GQAaccuracy: 35.19 (pos) -> 36.53 (pos+neg); neg only 36.77Improved
include explicit guidance 'Including any necessary facts or knowledge' in the rationale prompt (ablation)REVERIE-1.0-7b-lora · MMEscore: 1128.6 -> 1168.1Improved
add type-specific or length-control prompt constraints (ablation)rationale prompt · MMB, POPE, GQA, MMEscore: MME 1168.1 -> 1111.9 (type-specific); 1141.0 (plus length control)Hurt
apply consistency-based filtering of rationale data (ablation)REVERIE-1.0-7b-lora · MMEscore: 1132.3 -> 1168.1Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy