Research map / Vision-language and multimodal / Visual question answering
Tackling Vision Language Tasks through Learning Inner Monologues
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IMMO, where an LLM Reasoner and a VLM Observer converse in natural language (inner monologue), trained with supervised fine-tuning on GPT-3.5-augmented A-OKVQA data followed by alternating PPO, evaluated on ScienceQA and SNLI-VE.
Problem
Hybrid LLM-VLM systems that convert images to text are interpretable and cheap but hard to optimize end-to-end and underperform, while embedding-alignment VLMs need large data and are opaque; the paper targets vision-language reasoning (VQA and visual entailment) with a trainable interpretable multi
Approach
Model the LLM and VLM as a Reasoner and Observer that exchange multi-turn natural-language questions and answers, and jointly optimize them with a two-stage recipe (SL on synthesized inner-monologue dialogues, then alternating PPO with exact-match reward and KL penalty), so the system learns what to ask.
Key results
- IMMO SL+RL reaches 84.8% accuracy on ScienceQA test, 6.5 points above Vicuna-SL (78.3%) and above the 16-shot Vicuna CoT baseline (68.6%).
- On SNLI-VE dev, IMMO reaches 65.7% accuracy versus 55.3% for IdealGPT, while embedding-based OFA reaches 91.0%.
- Accuracy on ScienceQA rises from 0 to 2 conversation turns and then plateaus; few-shot performance degrades after 2 turns.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 16-shot inner monologue instead of 16-shot Chain-of-Thought | Vicuna-7B · ScienceQA test | accuracy: 68.6 to 74.0 | Helped |
| Train with two-stage SL+RL and inner monologue instead of Vicuna-SL with Chain-of-Thought | Vicuna-7B · ScienceQA test | accuracy: 78.3 -> 84.8 | Helped |
| Use IMMO instead of IdealGPT | vision-language reasoning on SNLI-VE · SNLI-VE dev | accuracy: 55.3% vs 65.7% | Helped |
| Increase inner-monologue turns from 0 to 2 (ablation) | IMMO few-shot and trained variants · ScienceQA test | accuracy | Helped |
| Use more than 2 inner-monologue turns (ablation) | Trained IMMO · ScienceQA test | accuracy | No clear effect |
| Use more than 2 inner-monologue turns (ablation) | Few-shot IMMO · ScienceQA test | accuracy | Hurt |
Surprising findings
- Few-shot inner monologue degrades after about 2 turns, suggesting untrained LLMs are less robust to noisy multi-turn conversation.
- The embedding-based OFA model outperforms all natural-language hybrid methods on SNLI-VE (91.0% vs 65.7%).
Limitations
- The Reasoner always asks a fixed number of turns rather than deciding when to stop.
- Only one Observer is used, and the synthetic SFT data was generated by GPT-3.5 rather than humans.
- Reward is exact match only and results are validated on just two tasks.
Experimental setup
Datasets: A-OKVQA ScienceQA SNLI-VE
Benchmarks: ScienceQA SNLI-VE
Metrics: accuracy
Baselines: PICa Vicuna 16-shot CoT Vicuna-SL IdealGPT MiniGPT4 LLaVA OFA UnifiedQA Chameleon
Builds on: PPO InstructGPT Chain-of-Thought prompting LoRA PICa
Closest papers by meaning
- Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models (ICML 2024 · similarity 0.90)
- Cross-modal Information Flow in Multimodal Large Language Models (CVPR 2025 · similarity 0.89)
- Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs? (ICML 2025 · similarity 0.89)
- VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning (ICLR 2025 · similarity 0.89)
- Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs (NeurIPS 2024 · similarity 0.89)
- Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.