Research map / Large language models / Language, safety and interpretability
Measuring Intent Comprehension in LLMs
Nadav Kunievsky, James Evans
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| scale up from small to large LLaMA/Gemma models | LLaMA and Gemma 3 models · Guesstimation tasks (24 tasks, 5 domains) | Intent Sensitivity: approximately 20% -> around 50% | Improved |
| scale up to the 70B model versus the 12B Gemma model | LLaMA 3.3 70B vs Gemma 3 12B · Guesstimation tasks (24 tasks, 5 domains) | Intent Comprehension Index | No clear effect |
| scale up model size | LLaMA and Gemma 3 models · Guesstimation tasks (24 tasks, 5 domains) | Articulation Sensitivity | Hurt |
| scale up model size | LLaMA and Gemma 3 models · Guesstimation tasks (24 tasks, 5 domains) | Meaningful Variability Share | Hurt |
| lower sampling temperature from 1 to 0.5 or 0.2 (ablation) | LLaMA and Gemma 3 models · Guesstimation tasks (24 tasks, 5 domains) | Model Uncertainty | Improved |
| scale up model size across topical domains | LLaMA 70B vs Gemma 27B · Guesstimation tasks (24 tasks, 5 domains) | Intent Sensitivity by domain | Mixed |
Closest papers by meaning
- Evaluating Large Language Models at Evaluating Instruction Following (ICLR 2024 · similarity 0.88)
- Uncovering Gaps in How Humans and LLMs Interpret Subjective Language (ICLR 2025 · similarity 0.87)
- Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.87)
- Potemkin Understanding in Large Language Models (ICML 2025 · similarity 0.87)
- Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs (ICLR 2025 · similarity 0.87)
- Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
