Research map / Large language models / Reasoning and chain of thought
CLAWS:Creativity detection for LLM-generated solutions using Attention Window of Sections
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLAWS, a white-box method that uses attention-weight ratios over prompt sections (guideline, problem, references, instruction, response) to classify math solutions as Hallucinated, Creative, or Typical, outperforming five uncertainty/attention baselines on 5 math RL models.
Problem
Detecting creativity in LLM mathematical solutions without human evaluation, distinguishing Creative solutions from Typical and Hallucinated ones, where prior work only addressed creativity in writing tasks or hallucination detection.
Approach
Divide the prompt into guideline, problem, reference solutions, and instruction sections, plus the generated response, and use the normalized average last-layer attention mass on each section as features for a classifier, rather than scalar uncertainty measures.
Key results
- CLAWS (prototype strategy) reaches F1_w 63.20 and F1_m 46.05 on the TEST set for Mathstral versus at most 49.86 and 35.96 for the best baseline on that set.
- For hallucination detection with Mathstral on TEST, CLAWS achieves F1_w 72.99 vs about 17 for baselines, which often predict a single class.
- Reported Cohen's kappa between the two LLM evaluators is 0.741.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CLAWS instead of the strongest threshold baseline | DeepSeek-Math-7B-RL creativity detection · TEST | macro F1: 35.77 -> 46.01 | Helped |
| use CLAWS instead of the strongest threshold baseline | Mathstral-7B creativity detection · TEST | macro F1: 28.89 -> 46.05 | Helped |
| use CLAWS instead of the strongest threshold baseline | OpenMath2-Llama3.1-8B creativity detection · TEST | macro F1: 35.96 -> 44.27 | Helped |
| use CLAWS instead of the strongest threshold baseline | OREAL-7B creativity detection · TEST | macro F1: 36.15 -> 40.18 | Helped |
| use CLAWS instead of the strongest threshold baseline | Qwen2.5-Math-7B creativity detection · TEST | macro F1: 45.29 -> 43.37 | Hurt |
| use CLAWS instead of the strongest threshold baseline | DeepSeek-Math-7B-RL hallucination detection · TEST | macro F1: 40.32 -> 67.24 | Helped |
| use CLAWS instead of the strongest threshold baseline | Mathstral-7B hallucination detection · TEST | macro F1: 25.47 -> 69.59 | Helped |
| use CLAWS instead of the strongest threshold baseline | OpenMath2-Llama3.1-8B hallucination detection · TEST | macro F1: 34.68 -> 64.70 | Helped |
| use CLAWS instead of the strongest threshold baseline | OREAL-7B hallucination detection · TEST | macro F1: 29.92 -> 56.36 | Helped |
| use CLAWS instead of the strongest threshold baseline | Qwen2.5-Math-7B hallucination detection · TEST | macro F1: 43.32 -> 53.30 | Helped |
| use last-layer attention instead of layer 5 attention (ablation) | CLAWS on DeepSeek-Math-7B-RL · TEST creativity detection | macro F1: 45.02 -> 46.01 | Helped |
| use last-layer attention instead of layer 5 attention (ablation) | CLAWS on Qwen2.5-Math-7B · TEST creativity detection | macro F1: 42.05 -> 43.37 | Helped |
Surprising findings
- Baseline uncertainty and hidden-state scores do not differ meaningfully across the three classes on the reference set, while attention-section ratios do.
- Hallucinated solutions concentrate attention on the Guideline section across all models.
Limitations
- Creativity is defined only within mathematical problem solving, and results may not transfer to other tasks.
- Labels come from LLM evaluators (o4-mini, Gemini-1.5-Pro) rather than human experts, and reference set is small (29 problems).
- Performance on OREAL is weak under the prototype strategy because of its high hallucination rate; white-box approach limits use to models of roughly 20B or less.
Experimental setup
Datasets: CreativeMath HARP
Benchmarks: CreativeMath test set HARP AMC HARP AIME HARP A(J)HSME
Metrics: weighted F1 macro F1 macro average precision AUROC
Baselines: Perplexity Logit Entropy Window Entropy Hidden Score Attention Score
Builds on: Perplexity-based uncertainty LLM-Check TabM XGBoost
Closest papers by meaning
- Assessing the Creativity of LLMs in Proposing Novel Solutions to Mathematical Problems (AAAI 2025 · similarity 0.87)
- IDEAFix: Evaluation Framework for Creative Defixation Prompting in LLMs (NeurIPS 2026 · similarity 0.87)
- Boosting of Thoughts: Trial-and-Error Problem Solving with Large Language Models (ICLR 2024 · similarity 0.87)
- CogMath: Assessing LLMs' Authentic Mathematical Ability from a Human Cognitive Perspective (ICML 2025 · similarity 0.87)
- Creativity or Brute Force? Using Brainteasers as a Window into the Problem-Solving Abilities of Large Language Models (NeurIPS 2025 · similarity 0.87)
- Let's Think Outside the Box: Exploring Leap-of-Thought in Large Language Models with Creative Humor Generation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.