Research map / Large language models / Reasoning and chain of thought

TRACE: Toulmin-based Reasoning Assessment through Constructive Elements for LLM CoT Evaluation

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Chain-of-thought training and analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TRACE, a reference-free metric that scores LLM chain-of-thought reasoning by labeling sentences with Toulmin and Flavell elements via a fine-tuned DeBERTa classifier and computing state validity and transition coherence.

Problem

Open-ended LLM evaluation lacks ground truth, and existing metrics (accuracy, perplexity, token length, lexical diversity) do not examine the logical structure of chain-of-thought reasoning.

Approach

Combine Toulmin's argumentation model and Flavell's metacognition theory into a sentence-level label scheme, classify CoT sentences with TRACE-DeBERTa, and score reasoning with a rule-based composite of allowed label states (state validity) and good/bad label transitions (transition coherence), which is also used as an RL reward.

Key results

• TRACE correlates with benchmark accuracy at Pearson r=0.741 (Spearman 0.755) across 39 benchmarks and 7 models, versus -0.147 for token length and -0.207 for MTLD.
• On GSM8K RL training of DeepSeek-R1-Distill-Qwen-1.5B, TRACE+Accuracy+Length reaches 84.69% vs 77.86% for Accuracy+Length and 74.75% base.
• TRACE-DeBERTa reaches macro F1 0.666 against three human annotators (Cohen's kappa 0.672); judge prediction accuracy on Arena Hard math is 64.37%.

Surprising findings

• TRACE-only reward caused CoT length to collapse, requiring accuracy reward to mitigate hacking.
• Per-benchmark sentence-length and token-length metrics showed negative correlation with accuracy, while the structural metric was strongly positive.

Limitations

• Measures argumentative structure rather than factual correctness, yielding false positives and false negatives.
• Prediction accuracy against LLM judges remains moderate (e.g., 55.78% coding, 50% creative writing).
• Relies on spaCy segmentation and a classifier with limited robustness on code and mixed-format text; transition sets were selected by correlation with accuracy.

Experimental setup

Datasets: GSM8K ARC-Challenge AIME MMLU MMLU-PRO GPQA SuperGPQA Arena-Hard-v2.0

Benchmarks: AIME GSM8K ARC MMLU MMLU-PRO GPQA SuperGPQA Arena-Hard-v2.0 ARC-Challenge

Metrics: Pearson correlation Spearman correlation accuracy prediction accuracy vs judge macro F1 Cohen's kappa

Baselines: Token Length Perplexity MTLD LLM-as-a-judge (GPT-4.1)

Builds on: Toulmin's argumentation model Flavell's metacognition theory GRPO DeBERTa-v3

Closest papers by meaning

• Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokens (NeurIPS 2026 · similarity 0.89)
• Reasoning Models Sometimes Output Illegible Chains of Thought (NeurIPS 2025 · similarity 0.88)
• The CoT Encyclopedia: Analyzing, Predicting, and Controlling how a Reasoning Model will Think (ICLR 2026 · similarity 0.88)
• Characterizing, Evaluating, and Optimizing Complex Reasoning (ICML 2026 · similarity 0.88)
• To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning (ICLR 2025 · similarity 0.88)
• Long Chain-of-Thought Reasoning Across Languages (ICLR 2026 · similarity 0.88)