Research map / Large language models / Retrieval and knowledge

A Comprehensive Evaluation on Event Reasoning of Large Language Models

AAAI 2025 · Large language models › Retrieval and knowledge · approach: Knowledge-augmented question answering · 8 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces EV² benchmark evaluating LLM event reasoning at schema and instance levels across causal, temporal, and hierarchical relations with CEC and CRR paradigms, finding imbalance and misalignment with human schema use, and that schema-knowledge guidance improves performance.

Problem

Event reasoning in LLMs has not been comprehensively evaluated across schema- and instance-level knowledge, diverse inter-event relations, and reasoning paradigms.

Approach

A two-level (schema and instance) benchmark built from synthesized and human-curated event graphs, with two paradigms (contextual event classification and contextual relation reasoning) across causal, temporal, and hierarchical relations, plus probes of whether LLMs use schema knowledge and whether explicit schema guidance helps.

Key results

• GPT-4 achieves 68.43 I-CEC and 63.81 I-CRR accuracy, far from satisfactory; open-source models like Llama2-7B and Vicuna-7B score under 40% on CEC.
• Average I-CEC is much higher than I-CRR (62.84 vs 53.58), and temporal relations are the hardest while causal relations are easiest.
• Explicitly providing schema events and relations in the prompt improves instance-level accuracy for all 11 models, e.g. Qwen2-7B CRR +12.11 and Vicuna-7B CEC +1.02 on average.

Measured effects reported in the paper

Surprising findings

• Only GPT-4 and GPT-4o produce correct event types when asked to articulate schema knowledge; other models fail to generate schema knowledge aligned with humans.
• Schema-level accuracy lags instance-level accuracy for most models, suggesting schema knowledge accumulation falls behind instance-level reasoning.

Limitations

• Benchmark size is modest (491 graph pairs; about 492-735 questions per task), and annotations were partly GPT-4 generated.
• Evaluation covers only 11 LLMs at 6-8B scale plus three GPT models, and only multiple-choice formats.

Experimental setup

Datasets: EECKG

Benchmarks: EV2

Metrics: accuracy F1-score

Baselines: GPT-4o GPT-4 GPT-3.5 Mistral-7B Qwen2-7B Baichuan2-7B Orca2-7B ChatGLM2-6B InternLM2-7B Llama2-7B Vicuna-7B

Builds on: EECKG ConceptNet

Closest papers by meaning

• TimE: A Multi-level Benchmark for Temporal Reasoning of LLMs in Real-World Scenarios (NeurIPS 2025 · similarity 0.88)
• Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning (ICLR 2025 · similarity 0.88)
• A State-Transition Framework for Efficient LLM Reasoning (ICLR 2026 · similarity 0.87)
• From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information? (ICML 2025 · similarity 0.87)
• LLM+AL: Bridging Large Language Models and Action Languages for Complex Reasoning About Actions (AAAI 2025 · similarity 0.87)
• GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (ICLR 2025 · similarity 0.86)