Research map / Large language models / Alignment and preferences
Reverse Engineering Human Preferences with Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a preamble generator with CoPG-based RL using judge-LLM scores as reward so that frozen candidate LLMs receive higher LLM-as-a-judge scores on MT-Bench, while evading perplexity and human detection.
Problem
LLM-as-a-judge evaluation is vulnerable to adversarial manipulation; prior attacks edit responses post hoc, which raises perplexity and is detectable. The paper asks whether upstream preambles optimised with RL can inflate judge scores without modifying the response text.
Approach
Reinforcement Learning for Reverse Engineering (RLRE): a trainable preamble generator is pipelined with a frozen candidate LLM, and the judge-LLM's numerical scores serve as rewards in a contrastive policy gradient (CoPG) loss, so the generator learns preambles that boost judge scores without directly observing them and without changing the response text.
Key results
- On MT-Bench, Command R7B candidate's overall Command R+ judge score rises from 7.29 (no attack) and 7.61 (best baseline, refinement) to 7.93 with tuned preambles.
- Human annotators labelled preamble-attacked responses 'attacked' only 14% of the time vs 12% for unattacked, while bandwagon, authority, refinement and universal attacks were flagged 95-100%.
- PPL-W false negative rate is at least 0.90 for verbosity, bandwagon and preamble attacks, versus 0.04 for the universal attack.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use RLRE preambles instead of the refinement-aware attack | Command R7B · MT-Bench overall | judge score: +0.32 points | Helped |
| use RLRE preambles instead of the refinement-aware attack | Command R · MT-Bench overall | judge score: +0.23 | Helped |
| use RLRE preambles instead of the universal adversarial attack | Llama 3.1 70B Instruct · MT-Bench overall | judge score: +0.05 | Helped |
| inject tuned preambles | Command R7B · Arena-Hard, Command R+ judge | Arena Score Rate: +3.5% | Helped |
| inject tuned preambles | Command R7B · Arena-Hard, GPT-4 judge | Arena Score Rate: +1.2 | No clear effect |
| remove the question from preamble-generator input (ablation) | Command R7B pipeline · MT-Bench overall | judge score: 7.93_{0.08} -> 7.76_{0.07} | Hurt |
| remove the instruction and input only special tokens (ablation) | Command R7B pipeline · MT-Bench overall | judge score: 7.93_{0.08} -> 7.70_{0.07} | Hurt |
| remove the question from preamble-generator input (ablation) | Command R pipeline · MT-Bench overall | judge score: 8.18_{0.05} -> 8.14_{0.08} | Hurt |
| remove the instruction and input only special tokens (ablation) | Command R pipeline · MT-Bench overall | judge score: 8.18_{0.05} -> 8.13_{0.13} | Hurt |
| remove the question from preamble-generator input (ablation) | Llama 8B+70B pipeline · MT-Bench overall | judge score: 8.22_{0.08} -> 8.19_{0.11} | Hurt |
| remove the instruction and input only special tokens (ablation) | Llama 8B+70B pipeline · MT-Bench overall | judge score: 8.22_{0.08} -> 8.18_{0.12} | Hurt |
| use preamble-based attack instead of no attack | Command R7B · MT-Bench math and reasoning responses | normalized exact-match accuracy: 45.9% -> 44.2% | Hurt |
Surprising findings
- Llama-generated preambles become largely non-fluent (repetitive or meaningless character sequences) yet still raise judge scores.
- Math accuracy does not improve under the attack (45.9% unattacked vs 44.2% attacked), suggesting judge gains come from layout rather than correctness.
Limitations
- Hyperparameters tuned only on the Command R7B+R7B pipeline and reused for others, so other pipelines may be under-optimised.
- Judge gains do not translate into accuracy gains on math and reasoning, and the score improvement may reflect layout rather than quality.
- Trained with a single judge via API; a jury of judges was not explored.
Experimental setup
Datasets: UltraFeedback MT-Bench Arena-Hard
Benchmarks: MT-Bench Arena-Hard
Metrics: judge score (1-10) PPL-W false negative rate human detection rate human quality rating normalised exact match accuracy
Baselines: Verbosity bias attack Bandwagon-effect bias attack Authority bias attack Refinement-aware bias attack Universal adversarial attack
Builds on: Contrastive Policy Gradient MT-Bench UltraFeedback Direct Preference Optimisation
Closest papers by meaning
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.90)
- Think-J: Learning to Think for Generative LLM-as-a-Judge (AAAI 2026 · similarity 0.89)
- Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.89)
- Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints (ICLR 2024 · similarity 0.89)
- Rule Based Rewards for Language Model Safety (NeurIPS 2024 · similarity 0.88)
- Preference Ranking Optimization for Human Alignment (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.