Research map / Large language models / Agents, code and math
Potemkin Understanding in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces a formal framework and two procedures (a three-domain benchmark and an automated self-judging lower bound) showing that LLMs frequently answer keystone questions correctly yet fail to apply the same concepts, a failure called potemkin understanding.
Problem
Benchmarks designed to test human conceptual understanding may not validly measure LLM understanding, because LLMs can answer keystone (definition) questions correctly while misinterpreting concepts in non-human ways, producing "potemkins".
Approach
Formalizes keystone sets and potemkin understanding, then measures potemkin rates as the failure to use a concept correctly conditioned on correctly defining it, plus a self-consistency (generate-then-classify) incoherence test and a lower-bound automated procedure using the LLM as its own judge.
Key results
- Models define concepts correctly 94.2% of the time, but potemkin rate on classification, generation, and editing averages 0.55, 0.40, and 0.40 respectively.
- Incoherence scores range from 0.02 to 0.64 across models; overall incoherence 0.22.
- The automated lower-bound potemkin rate on MMLU-derived questions is 0.62 overall.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate concept use after conditioning on a correct definition | Seven LLMs on the Potemkin Benchmark · Potemkin Benchmark—Classify | potemkin rate: 0.47–0.66 across models | Mixed |
| Evaluate concept use after conditioning on a correct definition | Seven LLMs on the Potemkin Benchmark · Potemkin Benchmark—Generate | potemkin rate: 0.23–0.62 across models | Mixed |
| Evaluate concept use after conditioning on a correct definition | Seven LLMs on the Potemkin Benchmark · Potemkin Benchmark—Edit | potemkin rate: 0.29–0.52 across models | Mixed |
| Compare a model's generated concept examples with its subsequent classifications | Nine LLMs · Incoherence evaluation | incoherence: 0.02 to 0.64 | Mixed |
| Use the same LLM to answer and judge generated follow-up questions | Nine LLMs on MMLU-based questions · MMLU-based automatic evaluation | potemkin rate (lower bound): Overall: 0.62 (0.01) | Mixed |
Surprising findings
- Expanding keystone sets with multiple classification questions yields only modest gains in concept understanding.
- Models often generate an instance of a concept and then fail to recognize their own output as valid, indicating internal incoherence.
Limitations
- Benchmark covers only 32 concepts across three domains and is not exhaustive.
- Automated procedure yields only a lower bound on potemkin rates.
- Many ground-truth labels were produced by paper authors or small expert panels, with small per-cell sample sizes.
Experimental setup
Datasets: MMLU
Benchmarks: Potemkin Benchmark (literary techniques, game theory, psychological biases) MMLU
Metrics: potemkin rate incoherence score accuracy
Baselines: GPT-o1-mini GPT-o3-mini
Builds on: MMLU
Closest papers by meaning
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.91)
- Truth Behind the Scene: Designing Evaluations Benchmarks to Assess LLMs’ Task-Specific Understanding over Test-Taking Strategies (AAAI 2025 · similarity 0.90)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.90)
- Improving Factuality and Reasoning in Language Models through Multiagent Debate (ICML 2024 · similarity 0.90)
- Calibrating Reasoning in Language Models with Internal Consistency (NeurIPS 2024 · similarity 0.90)
- Walk the Talk? Measuring the Faithfulness of Large Language Model Explanations (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.