Research map / Label-efficient and robust learning / Domain adaptation and generalization
NaRuto: Automatically Acquiring Planning Models from Narrative Texts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents NaRuto, a fully automated unsupervised system that extracts planning action models (STRIPS) from narrative texts by combining SRL-based event extraction with a BART-based commonsense predictor (COMET-BM) and textual similarity/contradiction filtering.
Problem
Automatically acquiring PDDL/STRIPS planning domain models from complex, colloquial narrative texts, which previous methods either handled only with human help or produced simplistic, story-mirroring models.
Approach
Builds action models by predicting preconditions and effects of extracted events with a COMET-style model fine-tuned on ATOMIC-2020 (COMET-BM, BART-based), then filters candidates for redundancy and contradiction using sentence-transformer similarity and NLI classifiers; also handles argument events and conditional events in SRL-based event structures.
Key results
- On the Aladdin domain, NaRuto(L) scored 3.34 average expert rating versus 2.81 for H2020 (fully automated) and 4.04 for semi-automated StoryFramer.
- On the Old American West domain, NaRuto(L) scored 3.57, above the semi-automated StoryFramer (3.26), while NaRuto(G) scored 2.98.
- In alternative plan generation on Aladdin, both NaRuto models solved 100% of 17 tasks with 8 distinct plans, while StoryFramer solved 1 of 30.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use restricted local negations instead of global negations (ablation) | NaRuto · Old American West | average score: 2.98 -> 3.57 | Helped |
| use restricted local negations instead of global negations (ablation) | NaRuto · Old American West | agreement: 43.0% -> 60.8% | Helped |
| use restricted local negations instead of global negations (ablation) | NaRuto · Old American West | disagreement: 39.3% -> 25.3% | Helped |
| use restricted local negations instead of global negations (ablation) | NaRuto · Aladdin | average score: 3.03 -> 3.34 | Helped |
| use restricted local negations instead of global negations (ablation) | NaRuto · Aladdin | agreement: 41.0% -> 52.0% | Helped |
| use restricted local negations instead of global negations (ablation) | NaRuto · Aladdin | disagreement: 38.3% -> 29.5% | Helped |
| use restricted local negations instead of global negations (ablation) | NaRuto · Old American West | solvable tasks: 4 (33.33%) -> 2 (16.66%) | Mixed |
| use restricted local negations instead of global negations (ablation) | NaRuto · Old American West | distinct plans: 1 -> 2 | Helped |
| use restricted local negations instead of global negations (ablation) | NaRuto · Aladdin | solvable tasks: 17 (100%) -> 17 (100%) | No clear effect |
| use restricted local negations instead of global negations (ablation) | NaRuto · Aladdin | distinct plans: 8 -> 8 | No clear effect |
| use NaRuto(L) instead of H2020 | narrative domain model generation · Old American West and Aladdin | average score: 2.54 -> 3.57; 2.81 -> 3.34 | Helped |
Surprising findings
- Restricted (local) negation effects scored consistently better than full (global) negation effects, suggesting positive effects and preconditions matter more than generated negations.
- Expert ratings of preconditions were consistently higher than those of effects across all generated models, indicating effect generation is harder.
Limitations
- Comparison systems were unavailable, so H2020 was approximated and baselines were partly taken from reported results.
- Evaluation uses only two short stories and nine experts with six responses per domain.
- Negative effect selection and story-context awareness remain challenges.
Experimental setup
Datasets: ATOMIC-2020 ATOMIC ConceptNet Aladdin story (Riedl and Young 2010) Old American West story (Ware 2014)
Benchmarks: Aladdin story Old American West story
Metrics: action name coverage expert Likert rating of precondition/effect appropriateness solvable tasks and distinct plans under modified initial sta
Baselines: StoryFramer H2020 Huo et al. (2020) Ground truth domains
Builds on: COMET ATOMIC-2020 StoryFramer AllenNLP SRL Sentence-BERT
Closest papers by meaning
- Large Language Models as Planning Domain Generators (Student Abstract) (AAAI 2024 · similarity 0.83)
- SeriesBench: A Benchmark for Narrative-Driven Drama Series Understanding (CVPR 2025 · similarity 0.82)
- Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks (ICML 2025 · similarity 0.82)
- Does Robin Hood Use a Lightsaber?: Automated Planning for Storytelling (AAAI 2024 · similarity 0.82)
- Narrative Action Evaluation with Prompt-Guided Multimodal Interaction (CVPR 2024 · similarity 0.82)
- Planning in the Dark: LLM-Symbolic Planning Pipeline Without Experts (AAAI 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.