Research map / Vision-language and multimodal / Multimodal LLMs

Training Data Efficiency in Multimodal Process Reward Models

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows MPRM training on VisualPRM400K saturates under random subsampling and proposes the Balanced-Information Score (BIS), a rollout-level selection criterion combining label mixture and MC-score reliability, so that 10% of the data matches full-data performance.

Problem

Training multimodal process reward models (MPRMs) requires large Monte Carlo-annotated corpora that are costly to train on; it is unclear how much of this data is needed and which rollouts are most informative.

Approach

Random subsampling of MC-annotated rollouts saturates quickly; informative rollouts are those mixing positive and negative steps (label mixture p(1-p)) whose positive steps have high average MC scores (reliability). BIS multiplies a smoothed mixture term by reliability, using only existing MC signals with no extra model calls. A teacher-student logistic analysis motivates the criterion.

Key results

• BIS-selected 10% subset on InternVL2.5-8B reaches 65.46 overall micro-F1, matching the full-data result (65.12) and beating Random-10% (62.86).
• On Qwen2.5-VL-7B, BIS-5% improves over Random-5% by about 10.9 points (64.42 vs 53.54) overall micro-F1.
• Best-of-16 reranking with MPRM trained on BIS-25% improves InternVL2.5-8B on MM-K12 from 33.13 to 41.00 over base, above Full-Data (39.00).

Surprising findings

• Lowest-MC-score subsets degrade performance due to pseudo-positive labels, and soft MC targets are clearly inferior to hard binary labels.
• BIS-25% best-of-N reranking matches or exceeds the full-data MPRM across four benchmarks.

Limitations

• Theory relies on strong simplifying assumptions (logistic teacher-student, bounded features, strong convexity) and is not validated on the actual MPRM.
• Only two backbones and one training dataset/benchmark family are evaluated.
• Performance peaks at moderate keep ratio and slightly drops afterward, with no principled rule for choosing the ratio.

Experimental setup

Datasets: VisualPRM400K-v1.1 VisualProcessBench

Benchmarks: VisualProcessBench MM-K12 OlympiadBench MathVerse MathVista

Metrics: micro-F1 macro-F1 best-of-N accuracy

Baselines: Random sub-sampling Full-Data training Mixed-25% Low-MC-25% Reliable-25% Soft labels

Builds on: process reward models VisualPRM400K

Closest papers by meaning

• Unlocking Multimodal Mathematical Reasoning via Process Reward Model (NeurIPS 2025 · similarity 0.87)
• Revisiting the Data Sampling in Multimodal Post-training from a Difficulty-Distinguish View (AAAI 2026 · similarity 0.86)
• Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Reward (ICLR 2026 · similarity 0.86)
• Learn to Think: Improving Multimodal Reasoning through Vision-Aware Self-Improvement Training (ICML 2026 · similarity 0.86)
• SCAN: Self-Denoising Monte Carlo Annotation for Robust Process Reward Learning (NeurIPS 2025 · similarity 0.86)
• MSRL: Scaling Generative Multimodal Reward Modeling via Multi-Stage Reinforcement Learning (CVPR 2026 · similarity 0.86)