Research map / Large language models / Language, safety and interpretability
LLM Targeted Underperformance Disproportionately Impacts Vulnerable Users
Elinor Poole-Dayan, Deb Roy, Jad Kabbara
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use low-education bio (Iran) instead of no bio | Claude 3 Opus · SciQ | accuracy: 95.60 -> 69.30 | Hurt |
| use low-education bio (Iran) instead of no bio | Claude 3 Opus · TruthfulQA | accuracy: 78.17 -> 66.22 | Hurt |
| use low-education bio instead of no bio | GPT-4 · SciQ and TruthfulQA | accuracy: 96.17 -> 95.40; 81.00 -> 80.17 | No clear effect |
| prepend bios on Llama 3 | Llama 3-8B · SciQ | accuracy: 88.7 -> 66.03 | Hurt |
| prepend bios on Llama 3 | Llama 3-8B · TruthfulQA | accuracy: 44.11 -> 50.06 | Improved |
| use high-education bio (Iran) vs control | Claude 3 Opus · SciQ | accuracy: 95.60 -> 91.30 | Hurt |
| use high-education Chinese bio vs control | Claude 3 Opus · TruthfulQA | accuracy: 78.17 -> 82.19 | Improved |
| use non-native-speaker or low-education bio | all three LLMs · TruthfulQA | accuracy | Hurt |
| combine low education with foreign origin | Claude 3 Opus · refusal rate | refusals: 3.61% -> 10.9% | Hurt |
| use low-education bios across countries | GPT-4 · SciQ | accuracy: 96.17 -> 93.10-94.10 | Hurt |
| education level isolated, adversarial vs non-adversarial TruthfulQA | GPT-4, Llama 3 · TruthfulQA | accuracy | Mixed |
Closest papers by meaning
- Measuring what Matters: Construct Validity in Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.88)
- Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyond (ICLR 2025 · similarity 0.88)
- Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs (ICLR 2024 · similarity 0.88)
- LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations (ICLR 2025 · similarity 0.88)
- Correlated Errors in Large Language Models (ICML 2025 · similarity 0.87)
- Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
