Research map / Large language models / Instruction and fine-tuning
Tailoring Self-Rationalizers with Multi-Reward Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MaRio, a multi-reward extension of Quark that distills small T5-large self-rationalizers from GPT-3 silver rationales, optimizing plausibility, diversity, consistency and task correctness to improve rationale quality and accuracy on five QA datasets.
Problem
Small language models (under 1B parameters) are assumed unable to self-rationalize well; prior work measures rationales only by downstream accuracy, ignoring intrinsic rationale quality such as plausibility, consistency and diversity.
Approach
Extends Quark's reward-conditioned control-token training to multiple rewards at once (Classic: all control tokens together; Additive: introduce reward tokens step-by-step), using automatic rewards for plausibility (Vera), consistency (reference-LM difference), n-gram diversity and task correctness.
Key results
- MaRio (Additive) on StrategyQA reaches 65.07 accuracy vs 57.64 for SFT, and Classic MaRio on NumerSense reaches 55.28 vs 46.23 for SFT.
- Human annotators preferred MaRio-only rationales for 83.15%, 75.3%, 71.49%, 67.44% and 66.6% of instances on StrategyQA, QuaRel, OpenBookQA, NumerSense and QASC respectively.
- A 0.7B T5-large trained with MaRio nearly matches GPT-3 (175B) on QuaRel and beats Flan-T5-XL/L and LLaMa-7B on several datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Multi-reward MaRio (Classic and Additive) vs. SFT on rationale quality and accuracy | T5-large · Five QA datasets | Avg. NRG: best MaRio variant above SFT on all five datasets | Helped |
| Adding task correctness as a reward and multi-reward training | T5-large (MaRio Additive on StrategyQA) · StrategyQA | Acc.: 57.64 to 65.07 | Helped |
| Single-reward Quark with the Product of rewards baseline | T5-large · StrategyQA | Avg. NRG: 58.66 to 59.75 | Mixed |
| Filtering to correct-answer rationales only (Filt-Acc) vs. MaRio on OpenBookQA | T5-large · OpenBookQA | Avg. NRG: 68.47 vs 68.64 for MaRio Classic | No clear effect |
| Filtering on all rewards thresholds (Filt-All) vs. Filt-Acc | T5-large · OpenBookQA | Acc.: 65.86 to 56.63 | Hurt |
| Single-reward Quark optimizing only task accuracy vs MaRio multi-reward (ablation) | T5-large · QuaRel | Plau.: single-reward Acc 0.74 vs MaRio Classic 0.77 | Hurt |
| Optimizing for reward-based Quark on factuality only (reward hacking) | T5-large · Qualitative reward-hacking examples | qualitative rationale quality | Hurt |
Surprising findings
- Optimizing for task accuracy alone (Filt-Acc, SFT) does not improve rationale quality, and rationales from task-only training were often worse than MaRio's on plausibility and consistency.
- Single-metric rewards led to reward hacking: factuality optimization induced repetitive or incomplete rationales even in larger models.
Limitations
- Rewards are automatic proxies (Vera, consistency model, n-gram diversity) and can be hacked; factuality and completeness lack reliable rewards.
- Human evaluation was limited to plausibility, consistency and overall preference, and only on MTurk.
- Relies on GPT-3 silver rationales for initialization and the SFT reference model.
Experimental setup
Datasets: StrategyQA QuaRel OpenBookQA NumerSense QASC
Benchmarks: StrategyQA QuaRel OpenBookQA NumerSense QASC
Metrics: task accuracy plausibility diversity consistency average normalized relative gain
Baselines: SFT Product of Rewards Filt-Acc Filt-All GPT-3 (text-davinci-003) Flan-T5 LLaMa
Builds on: Quark STaR Vera chain-of-thought prompting
Closest papers by meaning
- Fixing the Broken Compass: Diagnosing and Improving Inference-Time Reward Modeling (ICLR 2026 · similarity 0.88)
- Improving Factuality and Reasoning in Language Models through Multiagent Debate (ICML 2024 · similarity 0.87)
- Self-Aligned Reward: Towards Effective and Efficient Reasoners (ICLR 2026 · similarity 0.87)
- Learning Robust Rationales for Model Explainability: A Guidance-Based Approach (AAAI 2024 · similarity 0.87)
- Thinker: Learning to Think Fast and Slow (NeurIPS 2025 · similarity 0.87)
- Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice Questions (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.