Research map / Large language models / Language, safety and interpretability
The Confidence Trap: Gender Bias and Predictive Certainty in LLMs
Ahmed Sabir, Markus Kängsepp, Rajesh Sharma
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply Beta post-hoc calibration (ablation) | LLM pronoun probabilities · WinoBias | ECE: about three times lower ECE | Improved |
| apply Beta post-hoc calibration (ablation) | Llama-3.1-8B · WinoBias | accuracy vs human bias: 65.8% -> 74.9% | Improved |
| apply Beta post-hoc calibration (ablation) | Gemma-2-9B · WinoBias | accuracy vs human bias: 51.6% -> 54.7% | Improved |
| scale Gemma-2 from 9B to 27B (ablation) | Gemma-2 · WinoBias | ECE: 0.429 -> 0.366 | Improved |
| scale Gemma-2 from 9B to 27B (ablation) | Gemma-2 · WinoBias | Gender-ECE female: 0.156 -> 0.381 | Hurt |
| scale Llama-3.1 from 8B to 70B (ablation) | Llama-3.1 · WinoBias | ECE: 0.193 -> 0.216 | Hurt |
| replace occupation title with gender-neutral 'someone' (ablation) | GPT-J-6B · GenderLex | ECE: 0.076 -> 0.144 | Hurt |
| distill Llama-3.1-8B into DeepSeek-8B | Llama-3.1-8B · coreference tasks | Gender-ECE | Hurt |
| increase subset size from 50 to 500 (ablation) | ECE estimate · WinoBias, DeepSeek-8B | ECE std: 0.0381 -> 0.0076 | Improved |
| evaluate full free-text captions instead of single pronoun (ablation) | ECE · COCO captions, Gemma-3 | ECE: 0.178 - 0.240 clustered | No clear effect |
| compare female vs male pronouns | most LLMs · WinoBias/GenderLex | ECE: Gemma-2-9B male 0.067, female 0.895 | Hurt |
Closest papers by meaning
- Is Your Model Fairly Certain? Uncertainty-Aware Fairness Evaluation for LLMs (ICML 2025 · similarity 0.89)
- Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs (ICLR 2024 · similarity 0.88)
- Are Models Biased on Text without Gender-related Language? (ICLR 2024 · similarity 0.88)
- Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibrator (NeurIPS 2025 · similarity 0.87)
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (ICLR 2025 · similarity 0.87)
- Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
