Research map / Large language models / Alignment and preferences
K-Sort Eval: Efficient Preference Evaluation for Visual Generation via Corrected VLM-as-a-Judge
Zhikai Li, jiatong li, Xuewen Liu, Wangbo Zhao, Pan Du, Kaicheng Zhou, Qingyi Gu, Yang You, Zhen Dong, Kurt Keutzer
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add posterior correction to the VLM judge (ablation) | K-Sort Eval · Ablation T2I (FLUX.1-dev) | score: 29.32 -> 28.86 | Improved |
| add posterior correction to the VLM judge (ablation) | K-Sort Eval · Ablation T2V (CogVideoX-5b) | score: 31.86 -> 33.63 | Improved |
| add dynamic matching (ablation) | K-Sort Eval · Ablation T2I (FLUX.1-dev) | runs: 500 -> 81 | Improved |
| add dynamic matching (ablation) | K-Sort Eval · Ablation T2V (CogVideoX-5b) | runs: 300 -> 89 | Improved |
| add dynamic matching (ablation) | K-Sort Eval · Ablation T2I (FLUX.1-dev) | score: 28.79 -> 28.86 | No clear effect |
| add wrapping operation to the prompt (ablation) | K-Sort Eval · Ablation T2I (FLUX.1-dev) | score: 28.93 -> 28.86 | Improved |
| add wrapping operation to the prompt (ablation) | K-Sort Eval · Ablation T2V (CogVideoX-5b) | score: 33.55 -> 33.63 | No clear effect |
| add rule augmentation to the prompt (ablation) | K-Sort Eval · Ablation T2I (FLUX.1-dev) | score: 28.13 -> 28.86 | Improved |
| add rule augmentation to the prompt (ablation) | K-Sort Eval · Ablation T2V (CogVideoX-5b) | score: 33.10 -> 33.63 | Improved |
| add posterior correction to GPT-4o judge when correlating with human preference | GPT-4o VLM judge · 100 instance groups | correlation with human preferences | Improved |
| use GPT-4o judge instead of CLIP-based scorers | preference scoring · 100 instance groups | correlation with human preferences | Improved |
| distill SD-v3.5-large to turbo (4 steps) | SD-v3.5-large · Compressed models | score: 28.95 -> 27.71 | Hurt |
Closest papers by meaning
- K-Sort Arena: Efficient and Reliable Benchmarking for Generative Models via K-wise Human Preferences (CVPR 2025 · similarity 0.93)
- WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences (NeurIPS 2024 · similarity 0.88)
- RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data (AAAI 2025 · similarity 0.88)
- UVE: Are MLLMs Unified Evaluators for AI-Generated Videos? (NeurIPS 2025 · similarity 0.88)
- VHELM: A Holistic Evaluation of Vision Language Models (NeurIPS 2024 · similarity 0.88)
- VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
