Research map / Large language models / Language, safety and interpretability
Auto311: A Confidence-Guided Automated System for Non-emergency Calls
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds Auto311, a confidence-guided automated system that handles 311 non-emergency calls by predicting incident types hierarchically, itemizing report fields, and steering dialogue using Monte Carlo dropout confidence, evaluated on 11,796 Nashville call recordings.
Problem
Automating non-emergency (311) call handling, where callers may report multiple or shifting incident types and no fixed dialogue chart exists, so existing dialogue frameworks are inadequate.
Approach
A hierarchical, bootstrap-like cascade of binary classifiers predicts multiple incident types per call, with confidence defined as output consistency across Monte Carlo dropout trials, and confidence scores are used to fill report fields and reorder follow-up questions. Text-output confidence uses a combined keyword-overlap and SentenceBERT embedding score.
Key results
- Auto311 achieves an average F-1 of 92.54% (BERT backbone) on incident type prediction across 11 types, with confidence guidance improving lost/stolen F-1 from 91.50% to 93.00%.
- On information itemization, Auto311 reaches 0.9329 consistency on archived test data versus 0.6578 for GPT-4 and 0.6343 for GPT-3.5.
- Emulated calls are categorized with 94.49% mean accuracy and the system reduces follow-up turns as utterance size grows.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add confidence guidance to BERT-based incident type prediction | Auto311 · Lost Stolen incident type prediction | F-1: 91.50% -> 93.00% | Helped |
| Add confidence guidance to itemization (ablation) | Auto311 · Aggressive driver behavior details | consistency score: 0.8255 to 0.9164 | Helped |
| Add confidence guidance to itemization (ablation) | Auto311 · Incident location field | consistency score: 0.9035 -> 0.9631 | Helped |
| Add confidence guidance to itemization (ablation) | Auto311 · Caller name field | consistency score: 0.9478 -> 0.9962 | Helped |
| Add confidence guidance to itemization (ablation) | Auto311 · Caller phone field | consistency score: 1.0000 -> 1.0000 | No clear effect |
| Add confidence guidance to itemization (ablation) | Auto311 · Vehicle description field | consistency score: 0.8538 -> 0.9104 | Helped |
| Add confidence guidance to itemization (ablation) | Auto311 · Human/suspect description field | consistency score: 0.9678 -> 1.0000 | Helped |
| Add confidence guidance to itemization (ablation) | Auto311 · Property description field | consistency score: 0.9512 -> 1.0000 | Helped |
| Add confidence guidance to itemization (ablation) | Auto311 · Damaged property incident timing | consistency score: 0.9045 -> 1.000 | Helped |
| Add confidence guidance to itemization (ablation) | Auto311 · Aggressive driver behavior details | consistency score: 0.8255 -> 0.9164 | Helped |
| Add confidence guidance to itemization (ablation) | Auto311 · Check welfare relationship field | consistency score: 0.9023 -> 0.9855 | Helped |
| Add confidence guidance to itemization (ablation) | Auto311 · Minor crash traffic blockage (Y/N) | F-1: 80.0% -> 94.12% | Helped |
Surprising findings
- Off-the-shelf SQuAD-pretrained BERT scores only 0.2422 on itemization, far below GPT-3.5 and GPT-4 baselines.
- Traditional CNN classifiers score only about 63% average F-1 on the nine types shown, despite being competitive on some categories.
Limitations
- Evaluation uses data from a single call center (Nashville) and emulated rather than live multi-turn conversations.
- Handover control rules and sensitive-word lists are not exhaustive and the broader handover process is out of scope.
- Address validation and redundant or callback call handling are left to future work.
Experimental setup
Datasets: Nashville DEC non-emergency call recordings (11,796) SQuAD CUAD TriviaQA
Benchmarks: Nashville DEC non-emergency call recordings (archived and latest)
Metrics: F-1 precision recall accuracy consistency score
Baselines: LSTM CNN RCNN RNN Self-Attention Attention BERT GPT-3.5 GPT-4 DistilBERT-SQuAD2 BERT-SQuAD2 RoBERTa-SQuAD2
Builds on: BERT Monte Carlo Dropout SentenceBERT Whisper
Closest papers by meaning
- Sim911: Towards Effective and Equitable 9-1-1 Dispatcher Training with an LLM-Enabled Simulation (AAAI 2025 · similarity 0.82)
- SLS-Bench: A Benchmark for Incident Log Summarization with Synthetic Observability Data (NeurIPS 2026 · similarity 0.78)
- SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal (ICLR 2025 · similarity 0.78)
- DeEscalWild: A Real-World Benchmark for Automated De-Escalation Training with SLMs (NeurIPS 2026 · similarity 0.78)
- Towards Evaluating Proactive Risk Awareness of Multimodal Language Models (NeurIPS 2025 · similarity 0.77)
- AIR: Improving Agent Safety through Incident Response (ICML 2026 · similarity 0.77)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.