Research map / Large language models / Alignment and preferences
Anyprefer: An Agentic Framework for Preference Data Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Anyprefer synthesizes preference data via a cooperative two-player Markov game between a target model and a tool-augmented judge model, refined by prompt-level feedback, and releases the 58K-pair Anyprefer-V1 dataset.
Problem
Self-rewarding preference data generation is biased because the reward model shares weights with the target model, and manual or external-AI annotation is costly or misaligned with the target model's preferences.
Approach
Model preference data synthesis as a cooperative two-player Markov game in which the target model samples candidates and a judge model, augmented with external tools (detectors, captioners, segmenters, reward models), ranks them; a separate LLM reward model scores the resulting pairs and feedback optimizes the prompts of both players.
Key results
- Anyprefer raises LLaMA2 GSM8K accuracy from 22.44 to 38.14 and AlpacaEval 2.0 length-controlled win rate from 5.20 to 19.25.
- On LLaVA-1.5-7B, Anyprefer improves LLaVA-Bench from 63.4 to 69.2 and MM-Vet from 30.5 to 33.0 after three iterations.
- Human-judged accuracy of the judge model rises from 67.2% without tools/feedback to 89.6% with them.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add tool-augmented judgment (ablation) | Anyprefer · All applications | average benchmark score: 56.88 -> 59.88 | Helped |
| Add feedback-guided prompt optimization to tool-augmented judgment (ablation) | Anyprefer · All applications | average benchmark score: 59.88 -> 61.03 | Helped |
| Add tools (ablation) | Anyprefer · GSM8K | accuracy: 30.10 -> 37.53 | Helped |
| Add feedback (ablation) | Anyprefer with tools · GSM8K | accuracy: 37.53 -> 38.14 | Helped |
| Add tools and feedback (ablation) | Anyprefer · Natural vision-language understanding | MME cognition: 340.4 -> 362.9 | Helped |
| Optimize both target and judge models jointly rather than optimizing only one (ablation) | Anyprefer · Natural vision-language understanding | MME cognition: 350.2 -> 359.0 | Helped |
| Use rank3 + rank1 instead of rank3 + rank5 preference pairs (ablation) | Anyprefer · Natural vision-language understanding | LLaVA-W: 66.8 -> 68.5 | Helped |
| Add tools and feedback (ablation) | Anyprefer · Medical image analysis | IU-Xray ROUGE-L: 12.42 -> 23.69 | Helped |
| Jointly optimize target and judge models instead of only the target model (ablation) | Anyprefer · Medical image analysis | IU-Xray ROUGE-L: 14.87 -> 20.66 | Helped |
| Use rank3 + rank1 instead of rank3 + rank5 preference pairs (ablation) | Anyprefer · Medical image analysis | IU-Xray METEOR: 21.12 -> 25.47 | Helped |
| Continue iterative preference fine-tuning | Anyprefer with LLaMA2 · GSM8K | accuracy: 38.14 -> 39.04 -> 41.32 | Helped |
| Add tools and feedback to judge responses (ablation) | Natural vision-language judge model · Manual evaluation of 200 examples | judge accuracy: 67.2% -> 89.6% | Helped |
Surprising findings
- Using the target model's own weights as the reward model is biased; the paper reports tool-augmented judging and feedback as necessary to reach gains.
- Dispreferred-only synthesis is used for medical tasks because current Med-LVLMs cannot produce high-quality preferred responses.
Limitations
- Relies on closed GPT-4o as judge and reward model, so costs and reproducibility depend on API access.
- Medical task uses ground-truth as preferred responses, so the method does not synthesize both sides there.
Experimental setup
Datasets: GSM8K ARC-easy ARC-challenge AlpacaEval MME LLaVA-Bench MMBench MM-Vet ScienceQA VizWiz GQA POPE
Benchmarks: GSM8K ARC-easy ARC-challenge AlpacaEval 2.0 MME LLaVA-Bench MMBench MM-Vet ScienceQA VizWiz GQA POPE
Metrics: accuracy length-controlled win rate win rate BLEU ROUGE-L METEOR success rate
Baselines: self-rewarding meta rewarding VLFeedback POVID RLHF-V LLaVA-RLHF RT-1 Octo-small Octo-base OpenVLA-SFT
Builds on: direct preference optimization self-rewarding language models meta rewarding LLM-as-a-Judge Grounded SAM
Closest papers by meaning
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.89)
- Limited Preference Data? Learning Better Reward Model with Latent Space Synthesis (NeurIPS 2025 · similarity 0.89)
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.89)
- Preference Learning for AI Alignment: a Causal Perspective (ICML 2025 · similarity 0.89)
- 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.89)
- Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.