Research map / Vision-language and multimodal / Multimodal LLMs

CFPO: Counterfactual Policy Optimization for Multimodal Reasoning

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CFPO, an RL method that adds a counterfactual KL regularizer (suppressing high-saliency visual attention values) to GRPO and DAPO to enforce visual grounding in LVLM reasoning.

Problem

RL post-training of large vision-language models rewards only final-answer correctness, so models can ignore visual evidence in favor of language priors, hallucinate, or keep stale visual anchors during long chain-of-thought reasoning.

Approach

Within attention layers, identify high-saliency text-to-image attention entries via a mean+2std outlier mask, replace their values with the mean image-token value to form a counterfactual policy, and add a KL divergence between factual and counterfactual policies as a regularizer maximized during GRPO/DAPO training, without external reward models.

Key results

• CFPO-D raises DAPO average from 55.60% to 58.49% (+6.25% relative) on the main 10-benchmark suite.
• CFPO-G improves GRPO's average from 53.88% to 55.30% (+3.17% relative) and beats PAPO-G by 1.32% relative.
• On Qwen2.5-VL-7B, CFPO-G reaches 63.43 average versus 62.30 for PAPO-G.

Surprising findings

• Entropy loss is redundant for CFPO-G with GRPO's KL anchor but needed for CFPO-D, where removing the reference KL makes the counterfactual term harder to balance.

Limitations

• Training is about 50% slower per step due to attention-level interventions breaking fused kernels such as Flash-Attention.
• Main experiments use only 3B models, with limited evidence at larger scales.
• Causal claims rest on an approximated structural abstraction and a heuristic outlier mask, with no formal guarantee shown.

Experimental setup

Datasets: ViRL39K

Benchmarks: C-VQA-Real MARS-Bench POPE TextVQA MMMU-Pro(V) Geo3k We-Math MMK12 MathVerse LogicVista MathVista Counting

Metrics: average accuracy over 8 rollouts relative improvement

Baselines: GRPO DAPO PAPO-G PAPO-D

Builds on: GRPO DAPO PAPO Qwen2.5-VL

Closest papers by meaning

• Perception-Aware Policy Optimization for Multimodal Reasoning (ICLR 2026 · similarity 0.92)
• More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.91)
• PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshaping (NeurIPS 2026 · similarity 0.90)
• Reinforcing Multimodal Reasoning Against Visual Degradation (NeurIPS 2026 · similarity 0.89)
• Reasoning-Aligned Perception Decoupling for Scalable Multi-modal Reasoning (ICLR 2026 · similarity 0.89)
• Perceptual-Evidence Anchored Reinforced Learning for Multimodal Reasoning (CVPR 2026 · similarity 0.89)