Research map / Large language models / AI and society
Position: Principles of Animal Cognition to Improve LLM Evaluations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper proposing five evaluation principles drawn from animal cognition research and illustrating them with a transitive inference case study on GPT-4o.
Problem
Evaluating LLM capabilities is increasingly difficult because high benchmark scores may reflect shortcuts, brittle stimulus-specific behavior, or surface patterns rather than underlying cognitive capacities.
Approach
Import experimental design practices from comparative animal cognition (adversarial control conditions, stimulus robustness, failure-type analysis, mechanism-vs-behavior distinction, meeting the subject where it is) and apply them to LLM evaluation, demonstrated via classic transitive inference paradigms such as the n-term task.
Key results
- GPT-4o reached 100% accuracy on adjacent-pair transitive queries but only 40% on the trial-based n-term task, roughly chance, when non-adjacent pairs were tested.
- Accuracy in Experiment 1 correlated positively with symbolic distance across all six conditions, mirroring the symbolic distance effect seen in humans and animals.
- Replacing the ordinal operator and using random strings lowered performance from 1.00 to 0.92 and reverse-ranked animals to 0.78, showing sensitivity to prompt variation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Remove the ordinal operator and transitively linked elements, using an n-term trial-based task instead (ablation) | GPT-4o transitive inference evaluation · N-term trial-based vs. A vs. C | accuracy: 1.00 -> 0.40 | Hurt |
| Use reverse-ranked animal names instead of correctly size-ranked animal names (ablation) | GPT-4o transitive inference evaluation · Reverse ranked animal | accuracy: 0.97 -> 0.78 | Hurt |
| Replace ranked animal names with random strings (ablation) | GPT-4o transitive inference evaluation · Random string | accuracy: 0.97 -> 0.92 | Hurt |
| Remove transitive elements and the ordinal operator from the prompt (ablation) | GPT-4o transitive inference evaluation · Random string | accuracy: 1.00 -> 0.92 | Hurt |
Surprising findings
- GPT-4o performed at chance on novel non-adjacent pairs in the n-term task despite learning adjacent pairs within 3-5 trials.
- GPT-4o showed a symbolic distance effect analogous to humans and animals in Experiment 1.
Limitations
- Case study is a single model (GPT-4o) on a single reasoning capability, so generality of the principles is only illustrated, not validated.
- Principles are broad guidance and not operationalized into a formal protocol or metric.
- Authors acknowledge the methods may not yield results as interesting as anticipated.
Experimental setup
Metrics: accuracy symbolic distance effect
Builds on: match-to-sample paradigm n-term task Morgan's Canon
Closest papers by meaning
- Discovering Symbolic Cognitive Models from Human and Animal Behavior (ICML 2025 · similarity 0.87)
- ACED-Bench: Evaluating How LLM Agents Acquire and Act on Causal Evidence (NeurIPS 2026 · similarity 0.85)
- Does Spatial Cognition Emerge in Frontier Models? (ICLR 2025 · similarity 0.85)
- LLM Agents Can Be Choice-Supportive Biased Evaluators: An Empirical Study (AAAI 2025 · similarity 0.85)
- Position: Theory of Mind Benchmarks are Broken for Large Language Models (ICML 2025 · similarity 0.84)
- CogBench: a large language model walks into a psychology lab (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.