Research map / Large language models / Alignment and preferences
Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FIGA, an RL-free alignment method that builds a revised-response dataset (SPA) from Alpaca-7b outputs refined by ChatGPT and trains with token-level weights from Levenshtein edit operations to reward added tokens and penalize deleted ones.
Problem
SFT only imitates demonstrations and lacks fine-grained signals about what makes a response good or bad, while RLHF is complex and unstable; the paper seeks an RL-free alignment method with fine-grained quality signals.
Approach
Contrast a model's low-quality initial response with a minimally revised high-quality version (revised by ChatGPT using a reference demonstration), derive token-level encouraged/discouraged sets via Levenshtein distance, and apply weighted likelihood loss that increases probability of added/substituted tokens and decreases probability of deleted/substituted tokens.
Key results
- FIGA achieves average score 34.9 vs 31.1 for SFT (+3.2 gain) and 33.1 for PPO (85K), with reward 4.62 vs 4.56 for SFT.
- Removing revision drops the average from 34.9 to 31.1; removing reward filtering drops it to 32.1.
- Human evaluation: FIGA win rate against ChatGPT is 40%, against SFT 57%, against PPO (85k) 83%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FIGA instead of SFT | Alpaca-7b · SPA test set | Reward: 4.56 -> 4.62 | Helped |
| Use FIGA instead of PPO (85K) | Alpaca-7b · Overall | Average: 33.1 -> 34.9 | Helped |
| Add reward-based filtering to select low-performing cases (ablation) | FIGA · Overall | Average: 32.1 -> 34.9 | Helped |
| Revise initial responses rather than use reference responses (ablation) | FIGA · Overall | Average: 31.1 -> 34.9 | Helped |
| Use edit-based token identification rather than bag-of-words (ablation) | FIGA · Overall | Average: 32.7 -> 34.9 | Helped |
| Use edit-based token identification rather than ChatGPT weighted annotations (ablation) | FIGA · Overall | Average: 31.4 -> 34.9 | Helped |
| Use edit-based token identification rather than ChatGPT binary annotations (ablation) | FIGA · Overall | Average: 31.6 -> 34.9 | Helped |
| Select bad tokens with negative log-likelihood < 0.6 rather than ≥ 0.6 (ablation) | FIGA · Overall | Average: 29.3 -> 34.9 | Helped |
| Select bad tokens by the further-selection rule rather than penalize all edit-identified bad tokens (ablation) | FIGA · Overall | Average: 28.1 -> 34.9 | Helped |
| Penalize bad tokens by setting β=0.5 rather than β=0 (ablation) | FIGA · Overall | Average: 34.2 -> 34.9 | Helped |
| Set γ=0 rather than γ=0.3 (ablation) | FIGA · Overall | Average: 33.0 -> 34.9 | Helped |
| Use α=1 rather than standardized reward scores for token weights (ablation) | FIGA · Overall | Average: 33.4 -> 34.9 | Helped |
Surprising findings
- Using reward-score-based weights instead of the Levenshtein-based weighting lowers the average (33.4 vs 34.9), and penalizing bad tokens only when the model is confident (NLL < 0.6) matters.
- Training with the Instruction data pool yields higher average (32.8) than with the Preference pool (30.5) despite a lower reward score.
Limitations
- Evaluation reward model is the same as used in SPA construction and filtering, so reward scores may be biased toward it.
- Relies on ChatGPT for revisions and token scoring, so dependence on a proprietary model; small dataset of 17k and evaluations rely on GPT-based judging.
Experimental setup
Datasets: HH-RLHF ShareGPT Instruct GPT-J Pairwise SHP OpenOrca
Benchmarks: MMLU TruthfulQA CrowS-Pairs WinoGender Vicuna WizardLM
Metrics: reward score accuracy bias rate win rate average score
Baselines: SFT PPO CoH RRHF DPO Alpaca-7b ChatGPT
Builds on: RLHF PPO DPO RRHF CoH
Closest papers by meaning
- Alignment Dynamics in LLM Fine-Tuning (NeurIPS 2026 · similarity 0.90)
- The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (ICLR 2024 · similarity 0.90)
- Exploring Intrinsic Alignments Within Text Corpus (AAAI 2025 · similarity 0.90)
- Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment (AAAI 2025 · similarity 0.90)
- Preference Ranking Optimization for Human Alignment (AAAI 2024 · similarity 0.89)
- Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignment (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.