Research map / Large language models / Instruction and fine-tuning

Fine-Tuning Language Models to Know What They Know

NeurIPS 2026 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a bias-controlled d'_type2 measurement of LLM metacognition and ESMA, an evolution-strategy fine-tuning method with a joint correctness/meta-answer reward that substantially raises d'_type2 for Qwen2.5, Llama3.2 and Gemma3 models.

Problem

Measuring genuine metacognition (self-knowledge) in LLMs is confounded by response biases, heuristics, prompt-template effects and familiarity cues, so raw alignment scores can overstate metacognitive ability.

Approach

Use signal-detection d'_type2 under a dual-question protocol (independent direct and meta prompts) to isolate metacognitive sensitivity, and optimize the model with a gradient-free evolution strategy whose fitness is a joint reward over direct correctness and meta-answer alignment, which is hard to express as per-example backpropagation.

Key results

• ESMA raised d'_type2 for Qwen2.5 1.5B from 0.20 to 0.93 and for Qwen2.5 3B to 1.02, the highest reported value, versus 0.94 for GPT 5.2 and 0.95 for Claude Sonnet 4.5.
• On FictionalQA, ESMA increased d'_type2 for Qwen2.5 1.5B from 0.20 to 0.65 after SFT on fictional QA pairs.
• In the 1.5B ablation, ESMA (d'=0.93) outperformed SFT (0.40), PPO (0.31) and GRPO (0.70) in d'_type2; applying the top 10% of weight updates recovered roughly 80% of the gain.

Surprising findings

• Proprietary models with high accuracy and high raw alignment had lower d'_type2 than the fine-tuned 3B model, e.g. Gemini 3 Flash had d'_type2 0.68 despite 90.71% raw alignment.
• Applying only the top 10% of weight changes by magnitude recovered most of the metacognitive gain, with the remaining 50% of smallest updates contributing almost nothing.

Limitations

• Evaluation limited to factual short-answer retrieval (TriviaQA, FictionalQA); unclear transfer to complex reasoning.
• Performance remains well below human metacognitive standards, per the authors.
• Meta-question-based RL baseline (GRPO) required lower learning rate to avoid reward hacking in univariate settings; reward-hacking risk noted for alignment-only objectives.

Experimental setup

Datasets: TriviaQA FictionalQA

Benchmarks: TriviaQA FictionalQA

Metrics: d'_type2 raw alignment accuracy yes ratio YFR NFR Type 2 AUROC IDK alignment all alignment

Baselines: SFT PPO GRPO OpenAI GPT 5.2 Claude Sonnet 4.5 Gemini 3 Flash

Builds on: Evolution Strategies signal detection theory (meta-d') GRPO PPO

Closest papers by meaning

• Decoupling Metacognition from Cognition: A Framework for Quantifying Metacognitive Ability in LLMs (AAAI 2025 · similarity 0.87)
• metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.87)
• Tuning-Free Accountable Intervention for LLM Deployment – a Metacognitive Approach (AAAI 2025 · similarity 0.87)
• Dynamic Evaluation of Large Language Models by Meta Probing Agents (ICML 2024 · similarity 0.87)
• Can Large Language Models Derive High-Level Cognition from Low-Level and Fragmented Foundational Information? (AAAI 2025 · similarity 0.87)
• An Emulator for Fine-tuning Large Language Models using Small Language Models (ICLR 2024 · similarity 0.86)