Research map / Large language models / AI and society

Assessing the Creativity of LLMs in Proposing Novel Solutions to Mathematical Problems

AAAI 2025 · Large language models › AI and society · approach: Social-science view of AI evaluation · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces CreativeMath, an AoPS-sourced benchmark of math problems with multiple human solutions, and a multi-stage LLM-judged pipeline that measures whether LLMs can produce correct yet novel solutions given k reference solutions.

Problem

Existing math evaluations measure only the correctness of LLM solutions, not whether models can devise novel solution methods beyond provided human solutions.

Approach

Reframes math evaluation around creativity: given a problem and k known solutions, the LLM must produce a correct solution distinct from them, with novelty judged by LLM evaluators against reference solutions using explicit criteria (methodology, intermediate steps, assumptions, generalization, complexity), and split into known vs. unknown novelty relative to the remaining human solutions.

Key results

• Gemini-1.5-Pro achieved the highest correctness ratio (69.92%) and novelty ratio (66.94%), with N/C of 95.75%.
• Gemini-1.5-Pro reached 100% correctness at k=4, while novelty-to-correctness ratio decreased as more reference solutions were provided.
• Novelty-to-correctness ratio rose with problem difficulty (55.39% on AMC 8 to 83.01% on USAMO), while correctness fell from 71.80% to about 35%.

Measured effects reported in the paper

Surprising findings

• Novelty-to-correctness ratio increases with problem difficulty even as correctness drops.
• Math-specialized models (DeepSeek-Math-7B-RL, Internlm2-Math-20B) score high on MATH but have very low novelty ratios.

Limitations

• Novelty and correctness are judged by LLM evaluators rather than human experts, and the evaluators are LLMs possibly biased toward their own outputs.
• Evaluation uses a small subset of 400 problems and 605 samples, with sample sizes shrinking to 8 at k=4.
• Only greedy decoding and a 1024-token generation cap were used, and the paper notes possible noise at higher k.

Experimental setup

Datasets: CreativeMath (scraped from AoPS) MATH

Benchmarks: CreativeMath MATH

Metrics: correctness ratio novelty ratio novelty-to-correctness ratio novel-unknown ratio novel-unknown-to-novelty ratio

Baselines: GPT-4o Claude-3-Opus Gemini-1.5-Pro Llama-3-70B Qwen1.5-72B Yi-1.5-34B Mixtral-8x22B DeepSeek-V2 DeepSeek-Math-7B-RL Internlm2-Math-20B

Builds on: chain-of-thought prompting MATH GSM8K

Closest papers by meaning

• A Matter of Interest: Understanding Interestingness Judgments of Math Problems in Humans and Language Models (NeurIPS 2026 · similarity 0.91)
• CogMath: Assessing LLMs' Authentic Mathematical Ability from a Human Cognitive Perspective (ICML 2025 · similarity 0.91)
• OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalization (NeurIPS 2025 · similarity 0.90)
• Evaluating Large Vision-and-Language Models on Children's Mathematical Olympiads (NeurIPS 2024 · similarity 0.90)
• Creativity or Brute Force? Using Brainteasers as a Window into the Problem-Solving Abilities of Large Language Models (NeurIPS 2025 · similarity 0.89)
• Atomic Thinking of LLMs: Decoupling and Exploring Mathematical Reasoning Abilities (NeurIPS 2025 · similarity 0.89)