Research map / Large language models / Agents, code and math

SciReason: A Controllable Benchmark for Scientific Reasoning in LLMs

NeurIPS 2026 · Large language models › Agents, code and math · approach: Inductive and abstract reasoning evaluation

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SciR, a controllable benchmark for deduction, induction, and causal abduction over scientific documents, rendering formal tasks into domain-specific scientific genres with an invertibility check, and evaluates six LLMs under CoT, neuro-symbolic, and SymbCoT pipelines.

Problem

LLMs lack reliable evaluation on the three paradigmatic scientific inference types (deduction, induction, causal abduction) in realistic scientific documents, since human-annotated science benchmarks lack mechanistic ground truth and synthetic logic benchmarks do not resemble scientific text.

Approach

Generate tasks from formal objects (deduction trees, inductive rules, causal graphs) with verifiable answers, then render them into multi-document scientific discourse using domain-specific genres and an invertibility contract, so that inference difficulty and premise-obfuscation difficulty can be varied independently.

Key results

• gpt-4o NS accuracy on Hard drops from 88.6 on NL to 43.4 on obfuscated rendering, showing rendering hurts even neuro-symbolic pipelines.
• deepseek-r1 leads gpt-4o by 53 points on principled inference (NL CoT) versus 24 points on premise extraction on Easy averages.
• Both inference complexity and obfuscation reduce accuracy for all six models, with effects compounding (gpt-4o CoT 42.6 to 17.5 across NL Easy to Obf Hard).

Surprising findings

• Neuro-symbolic pipelines are still hurt substantially by obfuscated rendering even though the solver is sound.
• SymbCoT* performs far below CoT on induction tasks, possibly because Prolog-style formal output is unfamiliar to the LLM.

Limitations

• Each reasoning mode is instantiated by one paradigmatic case and one scientific surface, limiting generality.
• Single-run evaluation at temperature 0 with point estimates and no error bars; full evaluation costs about $1,500 in API calls.
• Synthetic tasks rarely resemble real scientific corpora, which lack a clean latent formal object.

Experimental setup

Datasets: DrugBank Sachs protein-signalling data

Benchmarks: SciR

Metrics: accuracy chance-normalised accuracy

Baselines: CoT neuro-symbolic (NS) SymbCoT*

Builds on: Prover9 Popper GIES SymbCoT Sachs et al. protein signalling network

Closest papers by meaning

• SciInstruct: a Self-Reflective Instruction Annotated Dataset for Training Scientific Language Models (NeurIPS 2024 · similarity 0.88)
• RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning (ICML 2025 · similarity 0.88)
• SciCode: A Research Coding Benchmark Curated by Scientists (NeurIPS 2024 · similarity 0.87)
• MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models (AAAI 2026 · similarity 0.87)
• SpecReason: Fast and Accurate Inference-Time Compute via Speculative Reasoning (NeurIPS 2025 · similarity 0.87)
• Putnam-AXIOM: A Functional & Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs (ICML 2025 · similarity 0.87)