Research map / Large language models / Language, safety and interpretability
In-Context Impersonation Reveals Large Language Models' Strengths and Biases
Leonard Salewski, Stephan Alaniz, Isabel Rio-Torto, Eric Schulz, Zeynep Akata
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the impersonated age from 2 to 20 | Vicuna-13B and Llama 2 · Two-armed bandit, ages 2–20 | average reward: β=0.17 for Vicuna-13B; β=0.26 for Llama 2 | Improved |
| Increase the impersonated age from 2 to 20 | Vicuna-13B · Two-armed bandit, ages 2–20 | directed exploration: β=-0.03 | Hurt |
| Increase the impersonated age from 2 to 20 | Vicuna-13B · Two-armed bandit, ages 2–20 | exploitation: β=0.04 | Improved |
| Increase the impersonated age from 20 to 60 | Vicuna-13B · Two-armed bandit | average reward | No clear effect |
| Impersonate a task expert instead of a non-domain expert | Vicuna-13B · MMLU | accuracy | Improved |
| Impersonate a domain expert instead of a non-domain expert | Vicuna-13B and ChatGPT · MMLU | accuracy | Improved |
| Use the official MMLU prompt instead of the paper's task formulation (ablation) | Vicuna-13B and ChatGPT with expert personas · MMLU | accuracy | Improved |
| Add a social-category prefix to expert personas (ablation) | Vicuna-13B and ChatGPT · MMLU | accuracy | Hurt |
| Impersonate an older person rather than a 2-year-old | ChatGPT · Stanford Cars | classification accuracy: ∼4% to 54.9% | Improved |
| Impersonate the topic expert rather than a non-expert | ChatGPT · CUB and Stanford Cars | classification accuracy | Improved |
| Impersonate a man rather than a woman | Vicuna-13B · Stanford Cars | classification accuracy | Improved |
| Impersonate a woman rather than a man | Vicuna-13B · CUB | classification accuracy | Improved |
Closest papers by meaning
- Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs (ICLR 2024 · similarity 0.88)
- Open Models, Closed Minds? On Agents Capabilities in Mimicking Human Personalities through Open Large Language Models (AAAI 2025 · similarity 0.86)
- Looking Inward: Language Models Can Learn About Themselves by Introspection (ICLR 2025 · similarity 0.85)
- Uncovering Gaps in How Humans and LLMs Interpret Subjective Language (ICLR 2025 · similarity 0.85)
- The Illusion of Role Separation: Hidden Shortcuts in LLM Role Learning (and How to Fix Them) (ICML 2025 · similarity 0.85)
- ToMATO: Verbalizing the Mental States of Role-Playing LLMs for Benchmarking Theory of Mind (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
