Research map / Large language models / Language, safety and interpretability
CHRONOS: A Schema-Based Event Understanding and Prediction System
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents CHRONOS, an IBM system for DARPA KAIROS that pairs an NLI-based zero-shot event extraction pipeline with complex event schema matching to instantiate schemas and make plausibility-judged predictions about news events.
Problem
Analyzing newsworthy events across multiple documents and predicting unobserved subevents with faithful, provenance-linked justifications, without training on domain-specific data.
Approach
Combines a zero-shot NLI text classifier (RoBERTa-Large fine-tuned on MNLI) used throughout the pipeline with symbolic complex event schemas, so that unmatched schema elements become predictions with schema-based justifications and source-sentence provenance, rather than LLM-generated explanations.
Key results
- Correct schema instantiated for 32 of 34 stories (100% in 6 of 7 domains; 50% in general terrorist attack).
- Event prediction plausibility is 100% in six domains; 78% in general terrorist attack.
- Argument plausibility ranges from 25% (coup) to 88% (hazardous spill) with the permissive filter; annotator agreement 88% with Cohen's kappa 0.77.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply a confidence threshold to predicted arguments (ablation) | CHRONOS · Complex Events Dataset | prediction plausibility (arguments) | Helped |
| Apply the Wikidata superclass filter to predicted arguments (ablation) | CHRONOS · Hazardous Spill and Riot | prediction plausibility (arguments) | Helped |
| Use the permissive filter (confidence threshold OR Wikidata superclass filter) (ablation) | CHRONOS · Complex Events Dataset | prediction plausibility (arguments) | Helped |
| Classify stories into the best-matching event schema | CHRONOS · Complex Events Dataset | schema accuracy: 32 out of 34 | Helped |
Surprising findings
- A simple top-level narrative classification via zero-shot NLI outperformed earlier bottom-up schema matching approaches.
- Predicted argument plausibility is much lower than event plausibility, mainly due to overly liberal argument matching.
Limitations
- Match recall is moderate (35-57% events), and argument plausibility is low in some domains.
- Only 34 stories evaluated with two annotators; general terrorist attack domain misclassified for 2 of 4 stories.
- Schema library creation is out of scope and not described in this paper.
Experimental setup
Datasets: KAIROS complex events dataset (34 stories, LDC) MNLI ACE05 MATRES KLUE Wikidata DARPA Wikidata (DWD)
Benchmarks: KAIROS complex events dataset
Metrics: schema instantiation accuracy event match recall argument match recall prediction plausibility (events) prediction plausibility (arguments)
Builds on: RESIN Wikidata PropBank MNLI zero-shot classification
Closest papers by meaning
- A Comprehensive Evaluation on Event Reasoning of Large Language Models (AAAI 2025 · similarity 0.84)
- From News to Forecast: Integrating Event Analysis in LLM-Based Time Series Forecasting with Reflection (NeurIPS 2024 · similarity 0.82)
- TimE: A Multi-level Benchmark for Temporal Reasoning of LLMs in Real-World Scenarios (NeurIPS 2025 · similarity 0.82)
- Logic Induced High-Order Reasoning Network for Event-Event Relation Extraction (AAAI 2025 · similarity 0.81)
- ChroKnowledge: Unveiling Chronological Knowledge of Language Models in Multiple Domains (ICLR 2025 · similarity 0.81)
- Is a Large Language Model a Good Annotator for Event Extraction? (AAAI 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.