Research map / Large language models / Agents, code and math
Agents' Room: Narrative Generation through Multi-step Collaboration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Agents' Room, a multi-agent collaborative framework where planning and writing agents share a scratchpad under a deterministic orchestrator, applied to long-form fiction, and releases the Tell Me a Story dataset and evaluation metrics.
Problem
Long-form story generation with LLMs is hard: plots, characters, and length are weak when generated in one pass, and existing prompting or planning approaches are brittle and under-specified.
Approach
Decompose narrative writing into specialized planning agents (conflict, character, setting, plot) and sequential writing agents (exposition through resolution) that communicate through a shared scratchpad, orchestrated deterministically; agents are fine-tuned on synthetic data created by backtranslating gold stories with a teacher LLM.
Key results
- Fine-tuned Agents' Room write and plan+write variants are preferred by expert human raters over single-pass baselines across plot, creativity, development, and language use.
- Human-written stories remain preferred overall over all machine systems, with a performance gap largest on plot and creativity.
- LLM evaluator agrees with humans at system level (Spearman rho=0.62, N=45) and the LLM prefers the same story in a second run 90.2% of the time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Agents' Room writing agents, with or without planning agents | Agents' Room · Tell Me a Story test set | human story preference | Helped |
| fine-tune the agents using synthetic back-translated training data | Agents' Room · Tell Me a Story test set | human story preference | Helped |
| use only planning agents with a single finalizer instead of the full plan-and-write setup (ablation) | Agents' Room · Tell Me a Story test set | human story preference | Hurt |
| include writing agents that generate the final story section by section | Agents' Room · Tell Me a Story test set | average words/story: 1,193 -> 3,006 | Helped |
| use writing agents without planning agents (ablation) | Agents' Room · Tell Me a Story test set | human story preference | Helped |
| use only zero-shot writing agents instead of fine-tuned writing agents (ablation) | Agents' Room · Tell Me a Story test set | human story preference | Hurt |
Surprising findings
- Plan-only Agents' Room variants underperform, attributed to the single finalizer agent being too simplistic to use planned elements.
- BertScore barely discriminates systems, preferring both the simplest baseline and the most complex system.
Limitations
- Tell Me a Story is small (123 training, 52 validation, 55 test examples), limiting fine-tuning and generalization.
- Orchestrator is a fixed deterministic sequence rather than learned, and narrative structure is restricted to one template.
- Machine stories still fall well short of human-written ones in plot and creativity.
Experimental setup
Datasets: Tell Me a Story WritingPrompts RocStories ChangeMyView WikiPlots
Benchmarks: Tell Me a Story test set
Metrics: pairwise human preference Bradley-Terry strength story length unique word ratio intra- and inter-story trigram repetition prompt trigram overlap Rouge-L BertScore Spearman correlation between LLM and human ratings
Baselines: zero-shot single-pass generation fine-tuned single-pass generation plan-then-write zero-shot self-reflection zero-shot decompose zero-shot 2Stage decompose
Builds on: self-reflection LoRA Bradley-Terry model
Closest papers by meaning
- StoryBox: Collaborative Multi-Agent Simulation for Hybrid Bottom-Up Long-Form Story Generation Using Large Language Models (AAAI 2026 · similarity 0.89)
- SceneDecorator: Towards Scene-Oriented Story Generation with Scene Planning and Scene Consistency (NeurIPS 2025 · similarity 0.84)
- NarrativePlay: An Automated System for Crafting Visual Worlds in Novels for Role-Playing (AAAI 2024 · similarity 0.83)
- ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate (ICLR 2024 · similarity 0.83)
- From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skills (NeurIPS 2026 · similarity 0.83)
- WritingBench: A Comprehensive Benchmark for Generative Writing (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.