Research map / Large language models / AI and society

Rethinking Psychometric Evaluation of LLMs: When and Why Self-Reports Predict Behavior

NeurIPS 2026 · Large language models › AI and society · approach: Critiques of LLM claims and judging · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Tests whether LLM self-reports predict behavior by contrasting Theory of Planned Behavior with Big Five, across within- vs. separate-session probing and parameter-grid vs. persona induction on 4 behavioral tasks and 11 LLMs.

Problem

Prior work found LLM self-report (Big Five) dissociates from behavior, but it is unclear whether this reflects LLMs lacking coherence or the use of a weakly predictive instrument and isolated sessions.

Approach

Replaces broad Big Five traits with task-specific Theory of Planned Behavior (TACT-anchored) items and manipulates session context (shared vs. separate) and identity induction (parameter grid vs. persona prompting), finding coherence is selective: strong within-session for TPB, collapsing across sessions for context-primed sycophancy but persisting for implicit bias and partly honesty.

Key results

• Within-session TPB self-report vs behavior pooled r=+0.40 excluding IAT (95% CI +0.37 to +0.43), matching human meta-analytic range; Big 5 best-construct r_aligned near zero (+0.06 to +0.07) on volitional tasks.
• Across separate sessions, sycophancy coherence collapses (r +0.47 to -0.07) while honesty partially survives (r +0.67 to +0.53); only 2 of 11 models retain significant separate-session coherence.
• Persona induction increases SR diversity and cross-session SR stability but yields zero models rescued in behavioral coupling.

Surprising findings

• Sycophancy behavior shows strong negative cross-session consistency for several models (e.g., Qwen 235B r=-0.96), indicating context-window priming rather than stable disposition.
• IAT explicit-implicit correlation is negative (r=-0.59), a compensatory-effort inversion where stronger anti-bias intention co-occurs with more stereotype-consistent responses.

Limitations

• Only four behavioral tasks and single-turn interactions limit generalization.
• Fixed snapshot of 11 model checkpoints; results may not persist across versions.
• Human psychometric instruments (TPB, BFI-44) adapted to LLMs; construct validity unverified.

Experimental setup

Datasets: PersonaHub

Benchmarks: Columbia Card Task Asch-style sycophancy paradigm two-stage confidence calibration honesty task text-based Implicit Association Test

Metrics: Pearson correlation Fisher-z pooled correlation Wilson CI Brier score flip rate d-score Cronbach's alpha

Baselines: Big Five Inventory

Builds on: Big Five Inventory Theory of Planned Behavior Columbia Card Task Asch conformity paradigm Implicit Association Test PersonaHub

Closest papers by meaning

• Tell me about yourself: LLMs are aware of their learned behaviors (ICLR 2025 · similarity 0.88)
• Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs (ICLR 2024 · similarity 0.87)
• Open Models, Closed Minds? On Agents Capabilities in Mimicking Human Personalities through Open Large Language Models (AAAI 2025 · similarity 0.86)
• Position: Rethinking LLM Bias Probing Using Lessons from the Social Sciences (ICML 2025 · similarity 0.86)
• Decision-Making Behavior Evaluation Framework for LLMs under Uncertain Context (NeurIPS 2024 · similarity 0.86)
• ToMATO: Verbalizing the Mental States of Role-Playing LLMs for Benchmarking Theory of Mind (AAAI 2025 · similarity 0.86)