Research map / Large language models / AI and society

A Matter of Interest: Understanding Interestingness Judgments of Math Problems in Humans and Language Models

NeurIPS 2026 · Large language models › AI and society · approach: Critiques of LLM claims and judging

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Compares LLM and human judgments of math problem interestingness (crowdsourced AMC raters and IMO competitors), finding LLMs align on mean ratings but not distributions or rationales, and that pooled models and filtered LLM-generated problems can help.

Problem

Whether LLMs judge math problem interestingness and difficulty in human-aligned ways, including distributions of judgments, rationales, reasoning effort, and ability to generate problems humans find interesting.

Approach

Directly measures LLM-human alignment on interestingness using two human populations (Prolific crowdworkers on AMC problems and IMO competitors), compares distributions via Wasserstein distance, tests pooling of multiple LLMs, and evaluates LLM-generated problems with human raters.

Key results

• Model-human R^2 on Prolific interestingness ranges from about 0.48 to 0.78, with Mistral family strongest; human split-half R^2 is 0.72.
• Only Mistral 7B Instruct (WD 12.4) approaches the human-human split-half distributional baseline (WD 9.5) among single models; a pooled set {Mistral-24B, Mistral-7B, Mixtral-8x7B, o3} reaches WD 9.07.
• After manual validity filtering, LLM-generated problems receive mean human interestingness around 59-63 across generators (Mistral 59.36, o3 62.46, Qwen 235B Thinking 62.84) with no significant difference.

Surprising findings

• Human interestingness and difficulty ratings are only moderately correlated (mean R^2 0.47), whereas reasoning models' interestingness and difficulty ratings are at or above 0.9 correlated.
• Models that best match humans on human-written problems are not significantly better at judging LLM-generated problems (r = -0.450, p = 0.165).

Limitations

• Small stimulus set of 18 Prolific problems and 8 variants, not a large-scale benchmark.
• Participants are crowdworkers with baseline math interest or IMO competitors, excluding beginners and experts in other settings.
• LLM-generated problem validity was filtered manually, not automatically.

Experimental setup

Datasets: AMC 8 and AMC 12 problems (with hand-written variants) IMO shortlist problems (2024 IMO survey)

Benchmarks: Prolific interestingness and difficulty ratings (18 problems) IMO 2024 survey interestingness rationales

Metrics: squared Pearson correlation (R^2) Wasserstein-1 distance Pearson correlation judgment reasoning token counts

Baselines: Human-Human split-half baseline Mistral 7B Instruct o3 GPT-5

Builds on: FunSearch AlphaTensor AlphaEvolve

Closest papers by meaning

• Assessing the Creativity of LLMs in Proposing Novel Solutions to Mathematical Problems (AAAI 2025 · similarity 0.91)
• CogMath: Assessing LLMs' Authentic Mathematical Ability from a Human Cognitive Perspective (ICML 2025 · similarity 0.91)
• RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics (NeurIPS 2025 · similarity 0.88)
• Evaluating Large Vision-and-Language Models on Children's Mathematical Olympiads (NeurIPS 2024 · similarity 0.88)
• OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalization (NeurIPS 2025 · similarity 0.87)
• Does GPT Really Get It? A Hierarchical Scale to Quantify Human and AI’s Understanding of Algorithms (AAAI 2025 · similarity 0.87)