Research map / Large language models / Language, safety and interpretability
Human-Guided Fair Classification for Natural Language Processing
Florian E. Dorner, Momchil Peychev, Nikola Konstantinov, Naman Goel, Elliott Ash, Martin Vechev
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Expand word-replacement constraints with style transfer and GPT-3 pairs (full C^e) used for CLP training | RoBERTa toxicity classifier · Civil Comments constraint test pairs | individual fairness on C^e: 78.5 -> 95.7 | Improved |
| Train with CLP on full C^e instead of baseline no CLP | RoBERTa toxicity classifier · Civil Comments constraint test pairs | balanced accuracy: 88.4 -> 85.0 | Hurt |
| Train on style-transfer constraints (C3) with CLP | RoBERTa toxicity classifier · Civil Comments constraint test pairs | generalization to WR50 and WR pairs: WR50 76.7 baseline style -> 95.3; WR 81.3 -> 97.1 (in bold) | Improved |
| Train with CLP on word-replacement constraints only (WR50 or WR) | RoBERTa toxicity classifier · Civil Comments constraint test pairs | individual fairness on ST (C3) pairs: CLP WR50 86.3 and CLP WR 90.5 on C3 vs CLP ST 95.4 and CLP Full 94.9 on C3 | Hurt |
| Human-guided filtering of C^e using learned similarity classifier (threshold 0.5) vs full C^e (ablation) | RoBERTa toxicity classifier · Human-validated test set T | individual fairness on T: 96.5 -> 97.4 (within noise, BA 85.9 both) | No clear effect |
| Filter adversarial constraint set C^e_adverse with learned similarity classifier (ablation) | RoBERTa toxicity classifier · Human-validated test set T | balanced accuracy: 71.1 -> 79.3 | Improved |
| Filter adversarial constraints with learned similarity classifier (ablation) | RoBERTa toxicity classifier · Human-validated test set T | individual fairness on T: 97.8 -> 98.7 | Improved |
| Relabel pairs likely mislabeled after active learning (add two extra labels) (ablation) | BERT similarity classifier phi-hat · Human-labeled test set T | balanced accuracy at t=0.5: 56.1 -> 62.0 | Improved |
| Relabel pairs likely mislabeled after active learning at threshold 0.1 (ablation) | BERT similarity classifier phi-hat · Human-labeled test set T | balanced accuracy: 59.4 -> 64.0 | Improved |
| Relabel at threshold 0.01 (no clear BA change) (ablation) | BERT similarity classifier phi-hat · Human-labeled test set T | balanced accuracy at t=0.01: 66.6 -> 66.7 | No clear effect |
| Add GPT-3 and style transfer pairs with word replacement to form C^e vs word replacement only | toxicity classifier constraints · Human evaluation of generated pairs | share of pairs judged should be treated similarly (unfair to treat differently): Word replacement 85.9 vs Style Transfer 85.2 vs GPT-3 83.2 (majority vote 97.5, 96.2, 93.7) | No clear effect |
| Use BERT concatenated-pair architecture instead of RoBERTa/DeBERTa for similarity classifier (ablation) | similarity classifier phi-hat · Word-replacement pairs, group transfer classification | balanced accuracy: BERT-Concat 86.7 vs RoBERTa-Concat 52.1 | Improved |
Closest papers by meaning
- Auto-Search and Refinement: An Automated Framework for Gender Bias Mitigation in Large Language Models (NeurIPS 2025 · similarity 0.85)
- All Should Be Equal in the Eyes of LMs: Counterfactually Aware Fair Text Generation (AAAI 2024 · similarity 0.84)
- Collapsed Language Models Promote Fairness (ICLR 2025 · similarity 0.84)
- FairImagen: Post-Processing for Bias Mitigation in Text-to-Image Models (NeurIPS 2025 · similarity 0.83)
- Arbitrariness and Social Prediction: The Confounding Role of Variance in Fair Classification (AAAI 2024 · similarity 0.82)
- FairNet: Dynamic Fairness Correction without Performance Loss via Contrastive Conditional LoRA (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
