Research map / Vision-language and multimodal / Multimodal LLMs
GenAlign: Towards Unified Alignment Framework of MLLMs via Generative Reward Model
Jingyu Zhang, Kun Yang, Ming Wen, Jiawei Zhao, Yuxuan Liu, Zhuoer Xu, shiwen cui
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GenAlign, a unified framework for aligning MLLMs with human preferences that builds on a generative reward model to address position bias and computational overhead.
Problem
Aligning multimodal large language models with human preferences; existing generative reward models suffer from severe position bias and high computational overhead.
Approach
Combines a robust generative reward model (designed to reduce position bias) with an alignment procedure in a unified framework, aiming to make reasoning-based reward modeling efficient for MLLMs. Specific mechanism is not stated in the abstract.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Online Position Debiasing (OPD) to GRM training (ablation) | GenAlign GRM · reward benchmarks | position gap / order-invariant accuracy | Improved |
| add bidirectional comparison to reward estimation (ablation) | GenAlign alignment · HaloQuest, LLaVA-1.5-7B | accuracy: 30.45 -> 45.16 | Improved |
| add bidirectional comparison (ablation) | GenAlign alignment · WildVision, Qwen2.5-VL-7B | win rate: 42.90 -> 45.00 | Improved |
| add advantage smoothing (ablation) | GenAlign alignment · WildVision, Qwen2.5-VL-7B | win rate: 43.80 -> 45.00 | Improved |
| add advantage smoothing (ablation) | GenAlign alignment · HaloQuest, LLaVA-1.5-7B | accuracy: 39.20 -> 45.16 | Improved |
| advantage smoothing instead of hard clipping (ablation) | GenAlign on Qwen2.5-VL-7B · HaloQuest | accuracy: 77.93 -> 81.99 | Improved |
| advantage smoothing instead of hard clipping (ablation) | GenAlign on Qwen2.5-VL-7B · MathVista | accuracy: 67.2 vs 67.4 | Mixed |
| increase number of reference responses K from 1 to 16 (all-to-all) (ablation) | GenAlign dynamic reference anchoring · MLLM benchmarks | performance | No clear effect |
| dynamic single-reference anchoring instead of exhaustive all-to-all comparisons (ablation) | GRM-based alignment · per-batch latency | latency (s): 320s -> 101s | Improved |
| increase inference samples N with majority voting / any-correct (ablation) | GenAlign GRM · VL-Reward and MM-RLHF benchmarks | accuracy | Improved |
| align with GenAlign reward instead of scalar RM | Qwen3-VL-8B · MathVista | accuracy: 72.6 -> 73.4 | Mixed |
| align with GenAlign reward | Llava-1.5-7B · HaloQuest | accuracy: 14.52 -> 45.16 | Improved |
Closest papers by meaning
- Generative RLHF-V: Learning Principles from Multi-modal Human Preference (NeurIPS 2025 · similarity 0.92)
- GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-Time Alignment (ICLR 2025 · similarity 0.92)
- MM-RLHF: The Next Step Forward in Multimodal LLM Alignment (ICML 2025 · similarity 0.91)
- Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback (ICML 2024 · similarity 0.91)
- RMB: Comprehensively benchmarking reward models in LLM alignment (ICLR 2025 · similarity 0.91)
- SEAL: Systematic Error Analysis for Value ALignment (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
