Research map / Large language models / Reasoning and chain of thought
OCEAN: Offline Chain-of-thought Evaluation and Alignment in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OCEAN, an offline chain-of-thought evaluation and alignment framework that uses a verbalized knowledge-graph policy (GPT2-Medium trained on Wikidata5M trajectories) to score LLM reasoning via a KG-IPS estimator and optimizes LLMs against it.
Problem
Offline evaluation and alignment of LLM chain-of-thought reasoning is underexplored; directly grounding LLM reasoning in knowledge graphs for feedback is hard due to entity linking and representation heterogeneity.
Approach
Model CoT generation as an MDP, use a verbalized knowledge-graph preference policy (small LM trained on GPT-4-verbalized Wikidata5M paths with QA-accuracy rewards) as a token-level behavior policy, and weight entity tokens with KG-based propensities and non-entity tokens with base-LLM propensities in an IPS estimator that is directly maximized for alignment.
Key results
- On Llama-3, OCEAN raises HotpotQA w/o context accuracy from 33.54% to 33.75% and improves estimated policy value from -10.35 to -8.10, while SFT drops to 16.49%.
- On commonsense tasks, OCEAN keeps average accuracy near base for Llama-3 (61.48 vs 61.56) and Phi-3.5 (65.44 vs 65.73), while SFT drops Llama-3 average to 51.04.
- On Gemma-2, OCEAN improves commonsense average from 57.10 to 62.22, while SFT falls to 30.58.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply OCEAN policy optimization | Llama-3 · HotpotQA, MuSiQue, and StrategyQA | estimated policy value: HotpotQA: -10.35 -> -8.10; MuSiQue: -9.90 -> -9.77; StrategyQA: -9.25 -> -5.53 | Helped |
| Apply OCEAN policy optimization | Gemma-2 · HotpotQA, MuSiQue, and StrategyQA | estimated policy value: HotpotQA: -31.88 -> -26.43; MuSiQue: -26.41 -> -22.15; StrategyQA: -14.06 -> -13.52 | Helped |
| Apply OCEAN policy optimization | Phi-3.5 · HotpotQA, MuSiQue, and StrategyQA | estimated policy value: HotpotQA: -19.49 -> -14.84; MuSiQue: -15.30 -> -13.47; StrategyQA: -13.87 -> -12.25 | Helped |
| Apply OCEAN policy optimization | Mistral-0.2 · HotpotQA, MuSiQue, and StrategyQA | estimated policy value: HotpotQA: -19.08 -> -3.12; MuSiQue: -6.40 -> -5.94; StrategyQA: -11.36 -> -11.21 | Helped |
| Apply OCEAN instead of Base | Gemma-2 and Mistral-0.2 · Commonsense reasoning average | accuracy: Gemma-2: 57.10 -> 62.22; Mistral-0.2: 60.98 -> 61.82 | Helped |
| Apply OCEAN instead of Base | Llama-3 and Phi-3.5 · Commonsense reasoning average | accuracy: Llama-3: 61.56 -> 61.48; Phi-3.5: 65.73 -> 65.44 | No clear effect |
| Apply OCEAN instead of Base | Llama-3 · HotpotQA (without context) | exact match: 33.54 -> 33.75 | Helped |
| Apply OCEAN instead of Base | Gemma-2, Phi-3.5, and Mistral-0.2 · HotpotQA (with context) | exact match: 26.33 -> 26.20; 32.13 -> 35.13; 26.82 -> 27.24 | Helped |
| Apply OCEAN instead of Base | All four backbones · knowledge-intensive tasks | accuracy | Mixed |
| Apply OCEAN instead of Base | Gemma-2 · AG News instruction tuning | accuracy: 47.03 -> 60.08 | Helped |
| Apply OCEAN instead of Base | Gemma-2 and Phi-3.5 · SST2, BoolQ, and AG News in-context learning | accuracy | Mixed |
Surprising findings
- SFT on knowledge graph data substantially degrades downstream commonsense and multi-hop performance for several backbones (e.g., Mistral-0.2 commonsense average falls from 60.98 to 31.05).
- OCEAN improves Gemma-2 and Mistral-0.2 commonsense performance even though it never trains on ground-truth answers.
Limitations
- Knowledge graph preference policy is a small GPT2-Medium model trained on only 6K GPT-4-verbalized pairs, limiting its fidelity.
- Reported gains are often small or mixed (e.g., many OCEAN results within ±1 point of base), and some tasks show declines.
- Variance and unbiasedness proofs rely on assumptions (bounded weights, sub-Gaussian rewards) that are loosely justified.
Experimental setup
Datasets: Wikidata5M CWQ
Benchmarks: HotpotQA MuSiQue StrategyQA ARC PubMedQA SciQA CSQA CSQA-2 CSQA-COT1000 OpenBookQA WinoGrande SST2
Metrics: exact match accuracy estimated policy value Self-BLEU Distinct-2 AlignScore
Baselines: SFT Base LLMs
Builds on: inverse propensity scoring DPO verbalized knowledge graph reasoning
Closest papers by meaning
- Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.89)
- Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs (NeurIPS 2024 · similarity 0.89)
- Making Slow Thinking Faster: Compressing LLM Chain-of-Thought via Step Entropy (ICLR 2026 · similarity 0.89)
- Generating Chain-of-Thoughts with a Pairwise-Comparison Approach to Searching for the Most Promising Intermediate Thought (ICML 2024 · similarity 0.89)
- Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph (ICLR 2024 · similarity 0.88)
- RATT: A Thought Structure for Coherent and Correct LLM Reasoning (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.