Research map / Large language models / Reasoning and chain of thought
First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MM-UPT, an unsupervised post-training method that runs GRPO on unlabeled multimodal math data with majority-vote pseudo-rewards, plus MLLM-generated synthetic questions, improving Qwen2.5-VL-7B on math benchmarks.
Problem
Post-training MLLMs with SFT or RL needs expensive annotated multimodal data; the paper asks whether an MLLM can keep improving its reasoning without any external labels or reward models, as a third post-training stage after SFT and RL.
Approach
Applies GRPO with self-rewarding: the majority-voted answer among G sampled responses serves as a pseudo-label, giving reward 1 to agreeing responses and 0 otherwise; extends this with in-context and direct self-synthesis of training questions.
Key results
- On Qwen2.5-VL-7B trained on MMR1 without labels, MathVista rose from 66.30 to 72.90 and We-Math from 62.87 to 68.74; average 49.47 to 53.17.
- Using synthetic direct-synthesized questions on GeoQA, average reached 52.67 versus 51.97 for original GeoQA questions.
- MM-UPT reduced pass@10 average from 0.8003 to 0.7250, indicating reduced response diversity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply MM-UPT on MMR1 with majority-vote rewards | Qwen2.5-VL-7B · MathVista | accuracy (%): 66.30 -> 72.90 | Helped |
| apply MM-UPT on MMR1 with majority-vote rewards | Qwen2.5-VL-7B · We-Math | accuracy (%): 62.87 -> 68.74 | Helped |
| train with direct synthetic questions instead of original questions | MM-UPT on Geometry3K · Math benchmarks (average) | Avg: 51.23 -> 52.32 | Helped |
| train with in-context synthetic questions instead of original questions | MM-UPT on Geometry3K · Math benchmarks (average) | Avg: 51.23 -> 51.00 | Hurt |
| train with direct synthetic questions instead of original questions | MM-UPT on GeoQA · Math benchmarks (average) | Avg: 51.97 -> 52.67 | Helped |
| train with synthetic questions instead of original questions | MM-UPT on MMR1 · Math benchmarks (average) | Avg: 53.17 -> 52.94; 53.17 -> 52.16 | Hurt |
| apply MM-UPT on Geometry3K (ablation) | MM-Eureka-7B · Math benchmarks (average) | Avg: 53.10 -> 53.78 | Helped |
| apply MM-UPT on Geometry3K (ablation) | ThinkLite-VL-7B · MathVista | accuracy: 69.00 -> 74.70 | Helped |
| apply MM-UPT on Geometry3K (ablation) | Qwen2.5-VL-3B · Math benchmarks (average) | Avg: 39.00 -> 41.72 | Helped |
| apply MM-UPT to difficult, low-competence training data | Qwen2.5-VL-7B · Math benchmarks (average), ThinkLite-11K | Avg: 49.47 -> 44.11 | Hurt |
| apply MM-UPT | Qwen2.5-VL-7B · Math benchmarks (pass@10 average) | pass@10: 0.8003 -> 0.7250 | Hurt |
| apply MM-UPT on MMR1 | Qwen2.5-VL-7B · ChartQA | accuracy: 71.96 -> 77.48 | Helped |
Surprising findings
- On the harder ThinkLite-11K dataset, MM-UPT degrades performance (average 49.47 to 44.11), since majority voting amplifies errors when the model lacks prior competence.
- Direct synthesizing of questions from images alone matched or exceeded training on original human-written questions on some datasets.
Limitations
- Majority voting fails when the base model has limited prior knowledge of the dataset.
- Pass@n drops, suggesting reduced diversity and exploration.
- Evaluation relies on a Qwen2.5-32B-Instruct judge and synthetic questions may contain hallucinations.
Experimental setup
Datasets: Geometry3K GeoQA MMR1 ThinkLite-11K DAPO-17K
Benchmarks: MathVision MathVerse MathVista We-Math ChartQA IconQA MATH Omni-MATH
Metrics: accuracy pass@10 average accuracy
Baselines: GRPO SFT (rejection sampling) LMSI SRLM Genixer STIC
Builds on: GRPO TTRL LMSI Self-Instruct EasyR1
Closest papers by meaning
- Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead (ICLR 2026 · similarity 0.91)
- UFT: Unifying Supervised and Reinforcement Fine-Tuning (NeurIPS 2025 · similarity 0.90)
- LLMs Can Evolve Continually on Modality for $\mathbb{X}$-Modal Reasoning (NeurIPS 2024 · similarity 0.89)
- RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.89)
- Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.89)
- Hybrid-LoRA: Bridging Full Fine-Tuning and Low-Rank Adaptation for Post-Training (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.