Research map / Reinforcement learning / RL from feedback
SOPO: Socratic Guided Policy Optimization for Span-Level Hallucination Detection
Jiawei Dong, Zilong Bai, Pengtian Zhu, Peng Zhou
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SOPO, a Socratic guided policy optimization approach that applies RLVR-style training to span-level hallucination detection to address localization ambiguity and reward sparsity.
Problem
Span-level hallucination detection in LLM outputs, where RLVR is hard to apply due to localization ambiguity and reward sparsity.
Approach
Applies Socratic guided policy optimization within an RLVR framework to tackle localization ambiguity and reward sparsity in span-level hallucination detection; specifics are not stated in the available abstract excerpt.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Socratic guidance with directive Direct-Feedback (ablation) | SOPO · RAGTruth span-level (overall) | span F1: 60.81 -> 63.31 | Improved |
| add Socratic guiding question to step-wise analysis (ablation) | SOPO feedback · RAGTruth span-level (overall) | span F1 | Improved |
| add step-wise analysis to guiding question (ablation) | SOPO feedback · RAGTruth span-level (overall) | span F1 | No clear effect |
| allow ground-truth leakage in teacher feedback (ablation) | SOPO · RAGTruth span-level (overall) | overall F1: overall F1 gap > 1.0 | Hurt |
| add Turn-based Advantage Scaling | SOPO · RAGTruth training | reasoning length per turn: stable ~1,700 tokens per turn vs GRPO inflation | Improved |
| use SOPO instead of GRPO | Qwen3-4B-2507 · RAGTruth span-level (overall, 10 runs) | span F1: 59.49 -> 63.34 | Improved |
| use SOPO instead of CAPO | Qwen3-4B-2507 · LLM-AggreFact (OOD) | mean balanced accuracy: 75.29 -> 75.93 | No clear effect |
| use SOPO instead of GRPO on OOD benchmarks | Qwen3-4B-2507 · HaluEval (OOD) | accuracy: 67.52 -> 69.77 | Improved |
| use FLOP-matched SOPO checkpoint (531 steps) instead of full SOPO (ablation) | SOPO · RAGTruth overall case-F1 | case F1: 83.72 vs GRPO 83.03 | No clear effect |
| scale teacher from 8B to 80B (ablation) | SOPO · RAGTruth span-level (overall) | span F1: 60.47 -> 63.31 | Improved |
| use SOPO instead of GRPO with 1.7B student (ablation) | Qwen3-1.7B · RAGTruth span-level (overall) | span F1: 55.8 -> 57.7 | Improved |
| use SOPO instead of GRPO with legacy 8B student (ablation) | Qwen3-8B · RAGTruth span-level (overall) | span F1: 60.6 -> 61.0 | Improved |
Experimental setup
Builds on: RLVR
Closest papers by meaning
- Learning to Reason for Hallucination Span Detection (ICLR 2026 · similarity 0.94)
- BALTO: Balanced Token-Level Policy Optimization for Hallucination Mitigation (NeurIPS 2026 · similarity 0.92)
- Token-Guard: Towards Token-Level Hallucination Control via Self-Checking Decoding (ICLR 2026 · similarity 0.91)
- Scalable Token-Level Hallucination Detection in Large Language Models (NeurIPS 2026 · similarity 0.91)
- Reasoning Models Hallucinate More: Factuality-Aware Reinforcement Learning for Large Reasoning Models (NeurIPS 2025 · similarity 0.90)
- Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Key (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
