Research map / Vision-language and multimodal / Multimodal LLMs
Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes a 7B vision-language model (LLaVA-v1.6-mistral-7b) end-to-end with PPO on task rewards, using chain-of-thought outputs and a CoT-aware log-likelihood scaling, on gym_cards and ALFWorld tasks.
Problem
Supervised visual instruction tuning does not efficiently yield decision-making agents for multi-step goal-directed tasks in interactive environments requiring visual recognition and language reasoning.
Approach
Treat the VLM's open-ended text output (CoT reasoning plus a text action) as the policy action, parse it into a legal environment action, estimate the action log-probability by summing action-token log-likelihood plus a scaled-down CoT log-likelihood, and fine-tune the entire VLM with PPO on environment rewards.
Key results
- On gym_cards the method reaches 45.5% average success vs 28.6% for Gemini, 25.4% for GPT4-V and 31.5% for CNN+RL (peak over training).
- On ALFWorld the method reaches 21.7% average success vs 19.4% for GPT4-V and 17.7% for LLaVA-sft.
- Removing CoT reasoning drops gym_cards average from 45.5% to 24.3% and ALFWorld average from 21.7% to 16.3%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune with RL | LLaVA-1.6-Mistral-7B · gym_cards average | episode success rate (%): 18.4% → 45.5% | Helped |
| fine-tune with RL | LLaVA-1.6-Mistral-7B · ALFWorld average | episode success rate (%): 17.7% → 21.7% | Helped |
| add CoT reasoning (ablation) | RL fine-tuned LLaVA-1.6-Mistral-7B · gym_cards / NumberLine | episode success rate (%): 26.9 → 89.4 | Helped |
| add CoT reasoning (ablation) | RL fine-tuned LLaVA-1.6-Mistral-7B · gym_cards / EZPoints | episode success rate (%): 29.9 → 50.0 | Helped |
| add CoT reasoning (ablation) | RL fine-tuned LLaVA-1.6-Mistral-7B · gym_cards / Points24 | episode success rate (%): 0 → 2.3 | Helped |
| add CoT reasoning (ablation) | RL fine-tuned LLaVA-1.6-Mistral-7B · gym_cards / Blackjack | episode success rate (%): 40.4 → 40.2 | Hurt |
| add CoT reasoning (ablation) | RL fine-tuned LLaVA-1.6-Mistral-7B · gym_cards average | episode success rate (%): 24.3 → 45.5 | Helped |
| add CoT reasoning (ablation) | RL fine-tuned LLaVA-1.6-Mistral-7B · ALFWorld average | episode success rate (%): 16.3 → 21.7 | Helped |
| use moderate CoT log-likelihood scaling (λ between 0.3 and 0.5) (ablation) | RL fine-tuning on NumberLine · gym_cards / NumberLine | episode success rate | Helped |
| use extreme CoT log-likelihood scaling (λ ≥ 0.7 or λ ≤ 0.1) (ablation) | RL fine-tuning on NumberLine · gym_cards / NumberLine | episode success rate | Hurt |
Surprising findings
- Without CoT reasoning, the method fails to improve NumberLine and EZPoints, indicating CoT is crucial for RL training of VLMs.
- Points24 remains near zero success for all methods, including the proposed one.
Limitations
- Performance gains limited by action space size and task difficulty; ALFWorld gains smaller than gym_cards.
- Points24 success remains near zero for all methods.
- Scaling factor lambda for CoT log-likelihood is sensitive; extreme values degrade performance.
Experimental setup
Datasets: gym_cards expert SFT data generated by task solvers ALFWorld GPT4-V-collected ALFWorld instruction data
Benchmarks: gym_cards (NumberLine, EZPoints, Points24, Blackjack) ALFWorld
Metrics: episode success rate episode return
Baselines: GPT4-V Gemini LLaVA-sft CNN+RL BUTLER BUTLER_g
Builds on: PPO LLaVA chain-of-thought prompting ALFWorld
Closest papers by meaning
- GFlowVLM: Enhancing Multi-step Reasoning in Vision-Language Models with Generative Flow Networks (CVPR 2025 · similarity 0.92)
- Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Models (NeurIPS 2025 · similarity 0.92)
- On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMs (ICML 2026 · similarity 0.91)
- Unveiling Chain of Step Reasoning for Vision-Language Models with Fine-grained Rewards (NeurIPS 2025 · similarity 0.91)
- VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use (ICLR 2026 · similarity 0.91)
- Code as Reward: Empowering Reinforcement Learning with VLMs (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.