Research map / Large language models / Alignment and preferences
The Devil Is in the Details: Tackling Unimodal Spurious Correlations for Generalizable Multimodal Reward Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies text-only (unimodal) spurious correlations in multimodal reward models and proposes a shortcut-aware training method that reweights samples using a text-only shortcut proxy, improving out-of-distribution generalization and Best-of-N downstream performance.
Problem
Multimodal reward models trained on existing preference datasets generalize poorly to out-of-distribution data because they rely on text-only shortcuts that hold only within the training distribution.
Approach
Train a text-only reward model as a shortcut proxy and use the ratio of its loss to the multimodal loss (Shortcut-Failure Coefficient) to dynamically upweight samples where text-only shortcuts fail, so the multimodal RM focuses on samples requiring visual grounding.
Key results
- Average out-of-distribution accuracy across six cross-distribution scenarios rises from 68.1 (standard MM-RM) to 78.5 with the shortcut-aware method, with in-distribution accuracy dropping slightly from 91.4 to 90.2.
- Best-of-64 on MM-Vet improves from 39.9 to 44.5 for the RLHF-V-trained RM and LLaVA-Bench from 73.1 to 79.6 for the same model.
- Shortcut-failure degradation averages 39.5 for standard MM-RMs; the shortcut-aware variant consistently reduces SFD across o.o.d. scenarios.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply SFC-based dynamic sample reweighting | InternVL2-8B MM-RM · Cross-distribution OOD average | accuracy: 68.1 to 78.5 | Helped |
| Apply SFC-based dynamic sample reweighting | InternVL2-8B MM-RM · Cross-distribution IID average | accuracy: 91.4 to 90.2 | Hurt |
| Apply SFC-based dynamic sample reweighting | InternVL2-8B MM-RM · Best-of-64 downstream benchmarks | score | Helped |
| Remove images during both training and testing (ablation) | Standard MM-RM · IID VLFeedback, POVID, and RLHF-V | accuracy: 87.1 -> 85.4; 100.0 -> 99.5; 87.0 -> 85.5 | Hurt |
| Remove images only during training (ablation) | Standard MM-RM · IID VLFeedback, POVID, and RLHF-V | accuracy: 87.1 -> 86.2; 100.0 -> 99.8; 87.0 -> 84.0 | Hurt |
| Remove images only during testing (ablation) | Standard MM-RM · IID VLFeedback, POVID, and RLHF-V | accuracy: 87.1 -> 82.5; 100.0 -> 98.8; 87.0 -> 84.3 | Hurt |
| Downsample VLFeedback training data from 79k to 3k instructions (ablation) | Standard MM-RM · VLFeedback-trained MM-RM, OOD average | accuracy: 79.3 to 78.8 | No clear effect |
| Downsample VLFeedback training data from 79k to 5k instructions (ablation) | Standard MM-RM · VLFeedback-trained MM-RM, OOD average | accuracy: 79.3 to 78.4 | No clear effect |
| Increase the maximum number of visual patches (ablation) | Standard MM-RM · OOD cross-distribution evaluation | accuracy | Helped |
| Scale InternVL2 model from standard to Shortcut-aware training | InternVL2-2B · Cross-distribution OOD average | accuracy: +6.4 | Helped |
| Scale InternVL2 model from standard to Shortcut-aware training | InternVL2-4B · Cross-distribution OOD average | accuracy: +6.8 | Helped |
| Scale InternVL2 model from standard to Shortcut-aware training | InternVL2-8B · Cross-distribution OOD average | accuracy: +10.4 | Helped |
Surprising findings
- A text-only RM achieves in-distribution accuracy comparable to the multimodal RM, and models trained multimodally still retain high text-only accuracy.
- Scaling the number of visual patches does not substantially reduce shortcut reliance, and VLFeedback-trained models generalize better even when downsampled.
Limitations
- Only tested on vision-language preference datasets focused on VQA and captioning, with InternVL2 backbones.
- Targets text-only shortcuts specifically; generalization to other spurious correlations is only discussed conceptually in the appendix.
- Trade-off of slight in-distribution accuracy decrease (91.4 to 90.2).
Experimental setup
Datasets: VLFeedback POVID RLHF-V
Benchmarks: MM-Vet LLaVA-Bench-in-the-Wild MMHal-Bench (MMHal-V)
Metrics: pairwise accuracy shortcut-failure degradation best-of-N score
Baselines: Standard MM-RM Text-only RM
Builds on: Bradley-Terry reward modeling group distributionally robust optimization best-of-N sampling
Closest papers by meaning
- BaseReward: A Strong Baseline for Multimodal Reward Model (ICLR 2026 · similarity 0.92)
- MM-RLHF: The Next Step Forward in Multimodal LLM Alignment (ICML 2025 · similarity 0.91)
- RRM: Robust Reward Model Training Mitigates Reward Hacking (ICLR 2025 · similarity 0.89)
- MSRL: Scaling Generative Multimodal Reward Modeling via Multi-Stage Reinforcement Learning (CVPR 2026 · similarity 0.88)
- RMB: Comprehensively benchmarking reward models in LLM alignment (ICLR 2025 · similarity 0.88)
- GRAM: A Generative Foundation Reward Model for Reward Generalization (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.