Research map / Large language models / Retrieval and knowledge
KOR-Bench: Benchmarking Language Models on Knowledge-Orthogonal Reasoning Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces KOR-Bench, a manually constructed benchmark of 1,250 rule-driven questions across five knowledge-orthogonal reasoning categories (operation, logic, cipher, puzzle, counterfactual), and evaluates ~40 chat and base LLMs on it.
Problem
Existing reasoning benchmarks rely heavily on domain-specific knowledge and memorized patterns, so they cannot cleanly measure whether models can apply newly defined rules to novel out-of-distribution problems.
Approach
Define "knowledge orthogonality" (rules independent of pretraining knowledge) and build a benchmark of 25 custom rules per category, each with 10 problems, so that solving depends on applying the newly stated rules rather than recalling facts.
Key results
- O1-Preview reaches 72.88% overall and O1-Mini 70.16%, versus Claude-3.5-Sonnet 58.96% and GPT-4o 58.00%.
- Best non-O1 models score at most 42.80% on Cipher and 16.80% on Puzzle, showing these categories are hard.
- Self-correction improves all models by an average of 10.36%, with most gains in the first two rounds.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use O1-Preview instead of GPT-4o | KOR-Bench evaluation · KOR-Bench | accuracy: 58.00 -> 72.88 | Helped |
| Use O1-Preview instead of GPT-4o | KOR-Bench evaluation · Cipher | accuracy: 42.80 -> 82.80 | Helped |
| Use O1-Preview instead of GPT-4o | KOR-Bench evaluation · Puzzle | accuracy: 16.80 -> 36.80 | Helped |
| Use Claude-3.5-Sonnet instead of GPT-4o | KOR-Bench evaluation · Logic | accuracy: 52.40 -> 67.20 | Helped |
| Use Qwen2.5-32B-Instruct instead of Qwen2.5-72B-Instruct | KOR-Bench evaluation · KOR-Bench | accuracy: 52.16 -> 54.72 | Helped |
| Add self-correction for up to two rounds | Models evaluated on KOR-Bench · KOR-Bench | accuracy: +10.36% | Helped |
Surprising findings
- Smaller Qwen2.5-32B-Instruct outperforms Qwen2.5-72B-Instruct in the overall score (54.72 vs 52.16).
- Base models in the counterfactual setting give real-life answers up to 36.8% of the time, indicating reliance on fixed knowledge.
Limitations
- Dataset is small (250 questions per category) and the authors plan to expand it.
- Results for proprietary models may be hard to reproduce due to API changes.
- Relies on manual rule design and LLM-assisted difficulty filtering, so the orthogonality is not formally verified.
Experimental setup
Datasets: KOR-Bench
Benchmarks: KOR-Bench
Metrics: accuracy real-life answer proportion (counterfactual) sub-problem pass rate
Baselines: O1-Preview O1-Mini Claude-3.5-Sonnet GPT-4o Meta-Llama-3.1-405B-Instruct Qwen2.5-72B-Instruct
Builds on: MMLU-Pro GSM8K Retrieval Head
Closest papers by meaning
- Reasoning-as-Logic-Units: Scaling Test-Time Reasoning in Large Language Models Through Logic Unit Alignment (ICML 2025 · similarity 0.87)
- From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information? (ICML 2025 · similarity 0.87)
- LINGOLY-TOO: Disentangling Reasoning from Knowledge with Templatised Orthographic Obfuscation (ICLR 2026 · similarity 0.87)
- LIMOPro: Reasoning Refinement for Efficient and Effective Test-time Scaling (NeurIPS 2025 · similarity 0.87)
- KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation (NeurIPS 2025 · similarity 0.86)
- Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.