AI Paper Atlas

Research map / Large language models / AI and society

Confirmation Bias: A Challenge for Scalable Oversight

Gabriel Recchia, Chatrik Singh Mangat, Jinu Nyachhyon, Mridul Sharma, Callum Canavan, Dylan Epstein-Gross, Muhammed Abdulbari

AAAI 2026 · Large language models › AI and society · approach: Data commons and human baselines · 2 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add LLM-generated research guidelines to open consultancy plus online research (ablation)open consultancy with gpt-4-0613 · Main study, both waves combinedAccuracy: 0.74 -> 0.73 (p = 0.19)No clear effect
add LLM-generated research guidelines to open consultancy (ablation)open consultancy with gpt-4-0613 · Main study, Wave 1Accuracy: 0.74 -> 0.72 (p = 0.027)Hurt
add LLM-generated research guidelines to open consultancy (ablation)open consultancy with gpt-4-0613 · Main study, Wave 2Accuracy: 0.74 -> 0.75 (p = 0.54)No clear effect
add LLM-generated research guidelines (ablation)open consultancy with gpt-4-0613 · Main study, Wave 1Logit of final probability for correct answer: 1.33 -> 1.05Hurt
add LLM-generated research guidelines (ablation)open consultancy with gpt-4-0613 · Main study, mixed modelGroup x timepoint interaction on logits: B = -0.16, OR = 0.85Hurt
add one-turn debate arguments to unstructured LLM interaction (ablation)gpt-3.5-turbo-0301 assisted participants · Preliminary experimentAccuracy: 69% -> 73% (p = 0.37 vs LLM alone at 75%)No clear effect
human-LLM interaction instead of LLM alonegpt-3.5-turbo-0301 · Preliminary experimentAccuracy: 75% -> 69% (p = 0.003)Hurt
conduct online research to verify the LLM consultant's answerparticipants evaluating gpt-4-0613 answers · Main study, mixed modelLogit of probability on correct answer: +1.17 logits (OR = 3.21)Improved
model answer is incorrect vs correctparticipants' updating towards the consultant · Main studyUpdate toward consultant after research: B = 0.72, OR = 2.06Mixed
more conversation turns with the LLM in Bowman et al. dataHuman + Model condition · MMLU reanalysisOdds of answering correctly: OR = 0.93 per additional turnHurt
more conversation turns with the LLM in Bowman et al. dataHuman + Model condition · QuALITY reanalysisAccuracyNo clear effect
resetting the conversation history at least onceHuman + Model condition · MMLU reanalysisProbability of answering correctly: chi2 = 4.24, p = 0.039Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy