Research map / Large language models / Instruction and fine-tuning
Exploring the LLM Journey from Cognition to Expression with Linear Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Defines cognitive capability (PCA probe on hidden states) and expressive capability (direct token accuracy) and tracks both across pretraining, SFT and RLHF in Baichuan-7B and Baichuan-33B.
Problem
Understanding how LLM capabilities develop across pretraining, SFT and RLHF, and why models internally encode answers yet fail to express them as token outputs.
Approach
Quantifies cognitive capability as the best per-layer accuracy of an unsupervised PCA direction on hidden states (RepE-style) and expressive capability as direct token-level accuracy, then tests their correlation statistically, offers a theoretical account via residual/pre-LN plateaus and vocabulary-layer KL dynamics, and evaluates optimization-free methods (few-shot, repeated sampling) to close t
Key results
- On RACE, expressive accuracy rises from 68.72% (SFT epoch 2) to 78.94% (SFT epoch 4) while cognitive accuracy stays around 74-80%.
- Correlation between cognitive and expressive accuracy is significant across all datasets (hypothesis-test probability <0.01% in most settings).
- Repeated sampling with 8 samples lets expressive capability match cognitive capability, per the authors.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune with SFT from Pretrain to SFT-2 | Baichuan-33B · HalluQA | Direct accuracy: 0.0607 -> 0.493 | Helped |
| continue SFT from SFT-2 to SFT-4 | Baichuan-33B · HalluQA | Direct accuracy: 0.493 -> 0.513 | Helped |
| apply PPO after SFT-4 | Baichuan-33B · HalluQA | Direct accuracy: 0.513 -> 0.563 | Helped |
| apply PPO after SFT-4 | Baichuan-33B · RACE | expressive accuracy: 78.94\% -> 77.01\% | Hurt |
| remove the 23-th intermediate layer (ablation) | Baichuan-7B · MMLU | accuracy: 0.5416 -> 0.5398 | No clear effect |
| remove the 23-th intermediate layer (ablation) | Baichuan-7B · CMMLU | accuracy: 0.5707 -> 0.5659 | No clear effect |
| use direct token generation instead of a linear SVM on hidden representations | Baichuan-33B · HalluQA | accuracy: 0.868 -> 0.0607 | Hurt |
| use few-shot examples | Baichuan-33B · single-choice benchmarks | expressive capability | Helped |
| use repeated sampling up to eight responses | Baichuan-33B · single-choice benchmarks | expressive capability | Helped |
| add the magical additional prompt | Baichuan-33B · single-choice benchmarks | expressive capability | No clear effect |
Surprising findings
- Deleting one intermediate layer (layer 23) of Baichuan-7B barely changes MMLU (0.5416 to 0.5398) and CMMLU (0.5707 to 0.5659).
- Direct token generation accuracy on HalluQA is 0.0607 at pretrain but the linear SVM reaches 0.868 on the same data.
Limitations
- Baichuan-33B is closed-source, limiting reproducibility.
- Cognitive capability quantified on single-choice tasks with a best-layer selection over test accuracy, and the reward model and annotation data are in-house.
- Theoretical analysis is heuristic and relies on approximations (e.g., independence assumptions in the residual-norm argument).
Experimental setup
Datasets: ARC-easy ARC-challenge CommonSenseQA OpenBookQA RACE HalluQA
Benchmarks: ARC-easy ARC-challenge CommonSenseQA OpenBookQA RACE HalluQA MMLU CMMLU
Metrics: accuracy consistency ratio binomial hypothesis-test probability KL divergence
Baselines: lm-evaluation-harness greedy likelihood scoring
Builds on: Representation Engineering (RepE) Proximal Policy Optimization lm-evaluation-harness
Closest papers by meaning
- Unveiling the Learning Mind of Language Models: A Cognitive Framework and Empirical Study (NeurIPS 2025 · similarity 0.88)
- Emergent Response Planning in LLMs (ICML 2025 · similarity 0.88)
- Can Large Language Models Derive High-Level Cognition from Low-Level and Fragmented Foundational Information? (AAAI 2025 · similarity 0.88)
- Generating Computational Cognitive models using Large Language Models (NeurIPS 2025 · similarity 0.88)
- CogBench: a large language model walks into a psychology lab (ICML 2024 · similarity 0.87)
- How does Architecture Influence the Base Capabilities of Pre-trained Language Models? A Case Study Based on FFN-Wider and MoE Transformers (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.