Research map / Large language models / Alignment and preferences
SoliReward: Mitigating Susceptibility to Reward Hacking and Annotation Noise in Video Generation Reward Models
Jiesong Lian, Ruizhe Zhong, Zixiang Zhou, Xiaoyue Mi, Long Hu, Yuan Zhou, Qinglin Lu, Yixue Hao, Junchi Yan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add win-tie pairs (BT-WT loss) to Bradley-Terry training (ablation) | SoliReward RM · HunyuanVideo post-training, VBench2 Human Fidelity | VBench2: 0.8693 -> 0.8999 | Improved |
| add win-tie pairs (BT-WT loss) (ablation) | SoliReward RM · HunyuanVideo post-training, VideoAlign MQ | MQ: 0.1719 -> 0.3302 | Improved |
| add win-tie pairs (BT-WT loss) (ablation) | SoliReward RM · Phy & Deform RM accuracy | RM accuracy (%): 77.63 -> 78.27 | No clear effect |
| replace Linear/token-logit/special-token head with HPQA multi-layer query attention (ablation) | SoliReward RM · Phy & Deform OOD | RM accuracy (%): 78.66 -> 80.08 (best alternative) | Improved |
| use HPQA instead of linear head (ablation) | SoliReward RM · TA (semantic alignment) OOD | RM accuracy (%): 31.92 -> 60.25 | Improved |
| use HPQA instead of special token + linear (ablation) | SoliReward RM · Phy & Deform OOD | RM accuracy (%): 73.61 -> 80.08 | Improved |
| add BCE penalty to BT-WT loss (ablation) | SoliReward RM · Phy & Deform | reward margin: 3.72 -> 2.97 | Hurt |
| add BCE penalty to BT-WT loss (ablation) | SoliReward RM · HunyuanVideo post-training | MQ: 0.3302 -> 0.1154 | Hurt |
| use single-item binary annotation instead of pairwise comparison (ablation) | annotation pipeline · Inter-annotator agreement | Krippendorff alpha: 0.3516 -> 0.4939 | Improved |
| use cross-prompt instead of in-prompt pairing (ablation) | SoliReward RM · RM accuracy | RM accuracy | No clear effect |
| guide DanceGRPO with SoliReward instead of VideoAlign MQ | HunyuanVideo · DanceGRPO post-training | VBench2 Human Fidelity: 0.8695 -> 0.8999 | Improved |
| SoliReward vs VideoPhy on semantic alignment OOD | reward model · TA OOD | RM accuracy (%): 60.52 -> 60.25 | No clear effect |
Closest papers by meaning
- Improving Video Generation with Human Feedback (NeurIPS 2025 · similarity 0.92)
- Aligning Human Sense: Calibrated Distributional Reward Learning for Video Generation (ECCV 2026 · similarity 0.91)
- Think, then Score: Decoupled Reasoning and Scoring for Video Reward Modeling (NeurIPS 2026 · similarity 0.89)
- SafeSora: Towards Safety Alignment of Text2Video Generation via a Human Preference Dataset (NeurIPS 2024 · similarity 0.88)
- Boosting Text-to-Video Generative Model with MLLMs Feedback (NeurIPS 2024 · similarity 0.88)
- Video-Bench: Human-Aligned Video Generation Benchmark (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
