Research map / Vision-language and multimodal / Multimodal LLMs
GFlowVLM: Enhancing Multi-step Reasoning in Vision-Language Models with Generative Flow Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes LLaVA-1.6-Mistral-7B as a non-Markovian GFlowNet policy with CoT-augmented action probabilities, outperforming SFT and PPO-based RL4VLM on NumberLine, BlackJack, and ALFWorld.
Problem
SFT assumes IID data and limits solution diversity, while PPO maximizes cumulative reward and under-explores, hurting multi-step visual reasoning and generalization in sequential decision-making with VLMs.
Approach
Treat a VLM as a GFlowNet forward policy over CoT-plus-action sequences, conditioned on full history (non-Markovian), trained with Var-TB, SubTB, or DB losses using task-based non-negative rewards so that it samples diverse high-reward solutions proportionally to reward; an SFT warm start supplies termination ([DONE]) probabilities.
Key results
- On NumberLine in-distribution, GFlowVLM variants reach 100.0% success vs. 89.4% for RL4VLM (Markovian).
- On ALFWorld, GFlowVLM w/ SubTB averages 26.1% success vs. 22.1% for RL4VLM (non-Markovian) and reaches 12.3% OOD vs. 6.1%.
- Off-policy ground-truth augmentation raises NumberLine OOD success from about 9% to 18.6% with DB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GFlowVLM DB instead of RL4VLM non-Markovian | NumberLine OOD · NumberLine OOD | success rate (%): 4.4 -> 9.1 | Helped |
| use GFlowVLM DB instead of RL4VLM non-Markovian | Blackjack · Blackjack | success rate (%): 41.0 -> 42.2 | Helped |
| use GFlowVLM SubTB instead of RL4VLM non-Markovian | ALFWorld · ALFWorld | average success rate (%): 22.1 -> 26.1 | Helped |
| use GFlowVLM SubTB instead of RL4VLM non-Markovian | ALFWorld · ALFWorld | Div@16: 1.11 -> 1.40 | Helped |
| add off-policy training data (ablation) | GFlowVLM DB · NumberLine OOD | success rate (%): 9.1 -> 18.6 | Helped |
| add off-policy training data (ablation) | GFlowVLM DB · Blackjack | success rate (%): 42.2 -> 43.8 | Helped |
| add SFT initialization (ablation) | GFlowVLM SubTB · NumberLine | success rate (%): 23.0 -> 100.0 | Helped |
| add SFT initialization (ablation) | GFlowVLM SubTB · Blackjack | success rate (%): 8.4 -> 41.7 | Helped |
| include history in the prompt (non-Markovian setting) (ablation) | GFlowVLM Var-TB · NumberLine | success rate (%): 93.4 -> 100.0 | Helped |
| include history in the prompt (non-Markovian setting) (ablation) | GFlowVLM Var-TB · NumberLine OOD | success rate (%): 4.7 -> 6.2 | Helped |
| include history in the prompt (non-Markovian setting) (ablation) | GFlowVLM Var-TB · Blackjack | success rate (%): 41.0 -> 41.4 | Helped |
| set the CoT weighting factor to 0.4 rather than 0.2 or 0.8 (ablation) | GFlowVLM · NumberLine | success rate | Helped |
Surprising findings
- Without SFT initialization, SubTB and DB perform very poorly on NumberLine and BlackJack (e.g., 23.0 and 8.4 on-policy), showing the termination-probability estimate needs a warm start.
Limitations
- Only small VLMs (7B) were used due to limited compute.
- Restricted to single-agent settings.
- GFlowNets require non-negative rewards, so reward functions were reshaped, making comparisons with RL4VLM's original rewards non-identical.
Experimental setup
Datasets: ALFWorld gym_cards NumberLine SFT dataset gym_cards BlackJack SFT dataset
Benchmarks: NumberLine NumberLine OOD BlackJack ALFWorld
Metrics: success rate Div@N diversity
Baselines: SFT RL4VLM PPO
Builds on: GFlowNets RL4VLM trajectory balance subtrajectory balance detailed balance
Closest papers by meaning
- Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning (NeurIPS 2024 · similarity 0.92)
- Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Models (NeurIPS 2025 · similarity 0.90)
- Unveiling Chain of Step Reasoning for Vision-Language Models with Fine-grained Rewards (NeurIPS 2025 · similarity 0.90)
- HopChain: Multi-Hop Data Synthesis for Generalizable Vision-Language Reasoning (NeurIPS 2026 · similarity 0.90)
- Synthesizing Multimodal Verifiable Game Data to Boost VLMs' General Reasoning (ICLR 2026 · similarity 0.90)
- All Roads Lead to Rome: Incentivizing Divergent Thinking in Vision-Language Models (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.