Research map / Large language models / Language, safety and interpretability
Decoupling Metacognition from Cognition: A Framework for Quantifying Metacognitive Ability in LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DMC, a framework that quantifies LLM metacognitive ability by fitting signal-detection-theory models to failure-prediction performance and decoupling it from cognitive ability, evaluated on eight MMLU/AGIEval/BBH-derived binary-choice datasets with GPT-3.5, GPT-4 and LLaMA2-70B.
Problem
Current metrics for LLM metacognition (e.g., calibration-based ECE, AUROC, Brier score) conflate metacognitive ability with task-level cognitive ability, so measured metacognition varies with task difficulty and is hard to interpret as a property of the model.
Approach
Model failure prediction with LLM-SDT (cognitive ability d and criterion c0) and LLM-Meta-SDT (adds confidence decision axes); recover cognitive ability from the answer-only confusion rates, then fit Meta-SDT under an optimal-metacognition constraint (confidence monotone with accuracy) and report the ratio d*/d as metacognitive ability.
Key results
- DMC's metacognition quantification varies less across datasets than calibration metrics (average CV 0.7263 for DMC vs 0.3992 ECE, 0.6253 AUROC, 0.0092 Brier for Verb-Vanilla; DMC is the lower-variability method in the reported table only for the compared metrics as listed).
- Self Random consistency elicitation gave the highest GPT-3.5 metacognition (average 0.5986) versus Verb-Vanilla (0.3225).
- Average metacognitive ability across three confidence methods: GPT-4 0.5491, GPT-3.5 0.4835, LLaMA2-70B 0.3210.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Verb-Topk instead of Verb-Vanilla confidence elicitation | DMC on GPT-3.5 · Eight datasets (average) | metacognitive ability: 0.3225 -> 0.5294 | Helped |
| Use Self Random instead of Verb-Vanilla confidence elicitation | DMC on GPT-3.5 · Eight datasets (average) | metacognitive ability: 0.3225 -> 0.5986 | Helped |
| Use Verb-Cot instead of Verb-Vanilla confidence elicitation | DMC on GPT-3.5 · Eight datasets (average) | metacognitive ability: 0.3225 -> 0.3768 | Helped |
| Use Perturbing instead of Self Random confidence elicitation | DMC on GPT-3.5 · Eight datasets (average) | metacognitive ability: 0.5986 -> 0.5599 | Hurt |
| Use Misleading instead of Self Random confidence elicitation | DMC on GPT-3.5 · Eight datasets (average) | metacognitive ability: 0.5986 -> 0.5606 | Hurt |
| Use GPT-4 instead of LLaMA2-70B with Verb-Vanilla confidence | DMC · Eight datasets (average) | metacognitive ability: 0.2505 -> 0.3636 | Helped |
| Use GPT-4 instead of LLaMA2-70B with Verb-Topk confidence | DMC · Eight datasets (average) | metacognitive ability: 0.3330 -> 0.5612 | Helped |
| Use GPT-4 instead of LLaMA2-70B with Self Random confidence | DMC · Eight datasets (average) | metacognitive ability: 0.3797 -> 0.7226 | Helped |
| Use DMC rather than calibration-based metrics to quantify metacognitive ability across datasets | GPT-3.5 · Eight datasets | CV | Mixed |
Surprising findings
- Verbalized confidence with chain-of-thought (Verb-Cot) gave only modest gains over vanilla verbalized confidence, and perturbing or misleading consistency methods scored slightly below Self Random.
Limitations
- Validated on only three LLMs (LLaMA2-70B, GPT-3.5, GPT-4) and binary-choice problems converted from multiple-choice questions.
- Metacognition-hallucination link is shown only via a qualitative trend in AbstainQA (Figure 4) rather than a rigorous causal test.
- Table 1 CV comparison is limited to GPT-3.5 and does not clearly establish that DMC is more stable than calibration metrics in the stated sense.
Experimental setup
Datasets: SAT Math (AGIEval) High School Mathematics (MMLU) CommonsenseQA Global Facts (MMLU) Boolean Expressions (BBH) Date Understanding (BBH) Professional Medicine (MMLU) Business Ethics (MMLU)
Benchmarks: SAT Math High School Mathematics CommonsenseQA Global Facts Boolean Expressions Date Understanding Professional Medicine Business Ethics AbstainQA
Metrics: DMC metacognitive ability (d*/d) coefficient of variation ECE AUROC Brier score Reliable Accuracy Abstain Accuracy
Baselines: ECE AUROC Brier score
Builds on: signal detection theory Xiong et al. 2024 confidence elicitation Tian et al. 2023 verbalized confidence Maniscalco and Lau 2012 meta-d'
Closest papers by meaning
- LLMs Show No Signs Of Individuated Metacognition (NeurIPS 2026 · similarity 0.90)
- Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations (NeurIPS 2025 · similarity 0.89)
- Tuning-Free Accountable Intervention for LLM Deployment – a Metacognitive Approach (AAAI 2025 · similarity 0.89)
- Fine-Tuning Language Models to Know What They Know (NeurIPS 2026 · similarity 0.87)
- Unveiling the Learning Mind of Language Models: A Cognitive Framework and Empirical Study (NeurIPS 2025 · similarity 0.86)
- MP: Endowing Large Language Models with Lateral Thinking (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.