Research map / Large language models / AI and society

Sim911: Towards Effective and Equitable 9-1-1 Dispatcher Training with an LLM-Enabled Simulation

AAAI 2025 · Large language models › AI and society · approach: Critiques of LLM claims and judging · 8 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Sim911 is an LLM-driven simulator that generates realistic, equitable 9-1-1 caller role-play for dispatcher training, combining knowledge bases built from archived calls, context-aware prompting (CoT, RAG, FSP), and looped validation checks.

Problem

Traditional 9-1-1 dispatcher training relies on labor-intensive human role-play that offers limited incident coverage and under-represents vulnerable caller groups; plain LLM simulators fabricate details and lack local grounding.

Approach

Builds tagged knowledge bases from archived 9-1-1 calls (incident specifications and caller images), injects them via RAG, CoT and few-shot prompting with dynamic backend selection, and filters responses with a looped validation co-pilot (format, alignment, factual checks) plus human feedback.

Key results

• Full Sim911 reaches PPL 11.07, METEOR 0.85, GMap 99.19%, and SAR 98.42%, outperforming the all-components-disabled GPT-4o baseline (PPL 61.99, GMap 61.01%, SAR 81.63%).
• User study with MNDEC personnel gave 4.89/5 average helpfulness for call-taking training, and 9 of 10 participants rated Sim911 on par with or better than human-led training.
• Validation threshold of 3 retries achieved a 98.47% negative sample capture rate at 61 ms/token latency.

Measured effects reported in the paper

Surprising findings

• Disabling the RAG sub-component caused a larger realism drop (PPL 57.19) than disabling knowledge construction in some metrics, and the GPT-4o-only system scored worst on realism and geocoding.
• Chain-of-thought removal improved some equity metrics (e.g., NRCLex 77.11 without CoT vs 73.31 full system), suggesting trade-offs between components.

Limitations

• Hallucinations still occur despite GPT-4o backend.
• Evaluation relies largely on a small user study (10 respondents) and deployment logs, with no released data or code.
• Knowledge base timeliness requires real-time updates that add overhead.

Experimental setup

Datasets: 11,841 archived 9-1-1 call recordings (Nov 2022 to May 2024) 2,641 past MNDEC calls for component-wise evaluation CAD logs and transcriptions

Benchmarks: Sim911 internal pre-configured runtimes (2,641 calls) real-world deployment logs (3,416 system and 3,409 user interactions)

Metrics: perplexity METEOR TTR Google Maps geocoding success rate (GMap) simulation alignment rate (SAR) BART score margin score NRCLex Gunning Fog index user Likert ratings

Baselines: GPT-4o without components (Sim911-All ablation) LLaMa 3.1

Builds on: retrieval-augmented generation chain-of-thought prompting few-shot prompting METEOR

Closest papers by meaning

• Auto311: A Confidence-Guided Automated System for Non-emergency Calls (AAAI 2024 · similarity 0.82)
• Multi-Agent Reinforcement Learning with Hierarchical Coordination for Emergency Responder Stationing (ICML 2024 · similarity 0.82)
• Can LLM Agents Respond to Disasters? Benchmarking Heterogeneous Geospatial Reasoning in Emergency Operations (NeurIPS 2026 · similarity 0.81)
• DeEscalWild: A Real-World Benchmark for Automated De-Escalation Training with SLMs (NeurIPS 2026 · similarity 0.81)
• AIR: Improving Agent Safety through Incident Response (ICML 2026 · similarity 0.78)
• Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs (NeurIPS 2024 · similarity 0.78)