AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

Human-Guided Fair Classification for Natural Language Processing

Florian E. Dorner, Momchil Peychev, Nikola Konstantinov, Naman Goel, Elliott Ash, Martin Vechev

ICLR 2023 · Large language models › Language, safety and interpretability · approach: Text generation and evaluation metrics · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Expand word-replacement constraints with style transfer and GPT-3 pairs (full C^e) used for CLP trainingRoBERTa toxicity classifier · Civil Comments constraint test pairsindividual fairness on C^e: 78.5 -> 95.7Improved
Train with CLP on full C^e instead of baseline no CLPRoBERTa toxicity classifier · Civil Comments constraint test pairsbalanced accuracy: 88.4 -> 85.0Hurt
Train on style-transfer constraints (C3) with CLPRoBERTa toxicity classifier · Civil Comments constraint test pairsgeneralization to WR50 and WR pairs: WR50 76.7 baseline style -> 95.3; WR 81.3 -> 97.1 (in bold)Improved
Train with CLP on word-replacement constraints only (WR50 or WR)RoBERTa toxicity classifier · Civil Comments constraint test pairsindividual fairness on ST (C3) pairs: CLP WR50 86.3 and CLP WR 90.5 on C3 vs CLP ST 95.4 and CLP Full 94.9 on C3Hurt
Human-guided filtering of C^e using learned similarity classifier (threshold 0.5) vs full C^e (ablation)RoBERTa toxicity classifier · Human-validated test set Tindividual fairness on T: 96.5 -> 97.4 (within noise, BA 85.9 both)No clear effect
Filter adversarial constraint set C^e_adverse with learned similarity classifier (ablation)RoBERTa toxicity classifier · Human-validated test set Tbalanced accuracy: 71.1 -> 79.3Improved
Filter adversarial constraints with learned similarity classifier (ablation)RoBERTa toxicity classifier · Human-validated test set Tindividual fairness on T: 97.8 -> 98.7Improved
Relabel pairs likely mislabeled after active learning (add two extra labels) (ablation)BERT similarity classifier phi-hat · Human-labeled test set Tbalanced accuracy at t=0.5: 56.1 -> 62.0Improved
Relabel pairs likely mislabeled after active learning at threshold 0.1 (ablation)BERT similarity classifier phi-hat · Human-labeled test set Tbalanced accuracy: 59.4 -> 64.0Improved
Relabel at threshold 0.01 (no clear BA change) (ablation)BERT similarity classifier phi-hat · Human-labeled test set Tbalanced accuracy at t=0.01: 66.6 -> 66.7No clear effect
Add GPT-3 and style transfer pairs with word replacement to form C^e vs word replacement onlytoxicity classifier constraints · Human evaluation of generated pairsshare of pairs judged should be treated similarly (unfair to treat differently): Word replacement 85.9 vs Style Transfer 85.2 vs GPT-3 83.2 (majority vote 97.5, 96.2, 93.7)No clear effect
Use BERT concatenated-pair architecture instead of RoBERTa/DeBERTa for similarity classifier (ablation)similarity classifier phi-hat · Word-replacement pairs, group transfer classificationbalanced accuracy: BERT-Concat 86.7 vs RoBERTa-Concat 52.1Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy