Research map / Large language models / Reasoning and chain of thought
Guiding LLM Decision-Making with Fairness Reward Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a process-level Fairness Reward Model on weakly LLM-labeled reasoning steps from BBQ and uses it to re-weight chain-of-thought answers, reducing equalized odds/opportunity gaps across COMPAS, Civil Comments, and Bias in Bios without hurting accuracy.
Problem
LLM decision-making with chain-of-thought sampling and majority voting can amplify social bias in high-stakes decisions, and prompting-based fairness mitigation is brittle.
Approach
A step-level fairness reward model (trained with binary cross-entropy on GPT-4o-mini bias labels for reasoning steps) scores each CoT step; chains are aggregated via a temperature-controlled softmax weighted vote that balances self-consistency against fairness, leaving the generator untouched.
Key results
- On Civil Comments-Religion, the equalized odds gap exceeds 60 points under CoT@1 and Maj@32 but falls by more than 10 points with FRM re-weighting, with accuracy rising about 4 points on Civil Comments settings.
- With Mistral-7B-Instruct-v0.3 as an unseen reasoner, FRM improves COMPAS accuracy by more than 10% and reduces Bias in Bios gender disparity by roughly 33%.
- GPT-4o-mini labels agree with human annotators on 75% of steps versus 88% human-human agreement (Cohen's kappa 0.23 vs 0.61 in appendix).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Reweight 32 CoT samples by FRM fairness scores | Llama-3.2-3B-Instruct on COMPAS · COMPAS | false-positive gap: 25-75% | Helped |
| Reweight 32 CoT samples by FRM fairness scores | Llama-3.2-3B-Instruct on Civil Comments · Civil Comments | group disparities: up to 40% | Helped |
| Reweight 32 CoT samples by FRM fairness scores | Llama-3.2-3B-Instruct on Bias in Bios · Bias in Bios | gender gap: more than 20% | Helped |
| Reweight CoT samples by fairness scores | Mistral-7B-Instruct-v0.3 on COMPAS · COMPAS | equalized odds gap | Helped |
| Reweight CoT samples by fairness scores | Mistral-7B-Instruct-v0.3 on COMPAS · COMPAS | accuracy: more than 10% | Helped |
| Reweight CoT samples by fairness scores | Mistral-7B-Instruct-v0.3 on Bias in Bios · Bias in Bios | gender disparities: roughly 33% | Helped |
| Use step-level process supervision with BBQ labels instead of LLM weak labels (ablation) | FRM · COMPAS and Civil Comments | fairness gaps | Hurt |
| Use chain-level outcome reward trained on LLM weak labels instead of step-level process reward (ablation) | FRM · COMPAS | fairness | No clear effect |
| Use chain-level outcome reward trained on LLM weak labels instead of step-level process reward (ablation) | FRM · Civil Comments | fairness: more than 10% | Hurt |
| Use chain-level outcome reward trained on BBQ labels (ablation) | FRM · COMPAS | fairness gaps | Hurt |
| Remove training and use a zero-shot prompted PRM (ablation) | FRM · COMPAS and Civil Comments | fairness and accuracy | Hurt |
| Reduce aggregation temperature from 0.8 to 0.4 and then 0.2 (ablation) | FRM · COMPAS, Civil Comments, and Bias in Bios | group outcome gaps: 0.8 to 0.4, and further to 0.2 | Helped |
Surprising findings
- Majority voting over 32 CoT samples can worsen fairness gaps (e.g., equalized odds on Civil Comments-Religion) relative to single CoT.
- Fairer reasoning sometimes increased accuracy, not only trading it off.
Limitations
- Relies on GPT-4o-mini weak labels that carry their own biases and show low kappa with humans (0.23 with GPT-4o-mini vs 0.61 between humans).
- Weights every reasoning step equally regardless of causal role in the final decision, causing failures where biased-but-ignored steps are penalized.
- English-only and US-centric; evaluated only on equalized odds and equalized opportunity.
Experimental setup
Datasets: BBQ COMPAS Civil Comments Bias in Bios
Benchmarks: BBQ COMPAS Civil Comments Bias in Bios
Metrics: accuracy equalized odds gap equalized opportunity gap
Baselines: CoT@1 Maj@32 Fairness Prompt ORM with BBQ labels PRM with BBQ labels ORM with LLM labels zero-shot PRM
Builds on: chain-of-thought prompting process reward models self-consistency
Closest papers by meaning
- Prompting Fairness: Integrating Causality to Debias Large Language Models (ICLR 2025 · similarity 0.87)
- Distributive Fairness in Large Language Models: Evaluating Alignment with Human Values (NeurIPS 2025 · similarity 0.87)
- JudgeBench: A Benchmark for Evaluating LLM-Based Judges (ICLR 2025 · similarity 0.87)
- Is Your Model Fairly Certain? Uncertainty-Aware Fairness Evaluation for LLMs (ICML 2025 · similarity 0.86)
- Matching Markets Meet LLMs: Algorithmic Reasoning with Ranked Preferences (NeurIPS 2025 · similarity 0.86)
- Any Large Language Model Can Be a Reliable Judge: Debiasing with a Reasoning-based Bias Detector (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.