AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

Truthfulness Despite Weak Supervision: Evaluating and Training LLMs Using Peer Prediction

Tianyi Alex Qiu, Micah Carroll, Cameron Allen

ICLR 2026 · Large language models › Language, safety and interpretability · approach: Uncertainty quantification for LLMs · 1 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
train with peer prediction DPO reward (0.135B expert) instead of LLM-as-a-Judge rewardLlama-3.1-8B deceptively finetuned · Free-form QA (37079 questions, 85 domains)ground-truth accuracyImproved
widen participant-to-expert size gap (smaller expert) (ablation)peer prediction evaluation · Honest vs deceptive detection (Llama-3.1-8B, Gemma2-2B, Gemma2-27B participants)cross-entropy of honesty regressionImproved
use LLM-as-a-Judge when judge is 5-20x smaller than deceptive participantLLM-as-a-Judge · Honest vs deceptive detectionhonesty prediction: worse than randomHurt
increase number of participants (ablation)peer prediction · Llama-3.1-8B participants, SmolLM-360M experthonesty predictionImproved
increase number of experts (ensemble) (ablation)peer prediction · Honest vs deceptive detectionhonesty regression R^2 surplusImproved
ensemble weight exponent alpha deviating from -1 (ablation)expert ensemble · Honest vs deceptive detectionsurplusHurt
increase expert size from 0.5B to 7B (ablation)peer prediction · Correctness prediction (pooled)Pearson correlation: 0.1947 -> -0.1595Hurt
increase judge size from 135M to 7B (ablation)LLM-as-a-Judge · Correctness prediction (pooled)Pearson correlation: 0.0352 -> 0.2379Improved
make deceptive participants half or more of population (ablation)peer prediction · 4 Llama-3.1-8B participants, Mistral-7B experthonest vs deceptive scoresHurt
use peer prediction with a small SmolLM-360M expert under majority deception (ablation)peer prediction · 4 Llama-3.1-8B participants, SmolLM-360M expertdeception detectionImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy