Research map / Large language models / Language, safety and interpretability
NATURE: Natural Auxiliary Text Utterances for Realistic Spoken Language Evaluation
David Alfonso-Hermelo, Ahmad Rashid, Abbas Ghaddar, Philippe Langlais, Mehdi Rezagholizadeh
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply one random NATURE operator per utterance | Stack-Prop+BERT evaluation · ATIS, SNIPS, NLU-ED (Avg.) | E2E accuracy: 80.6 -> 59.0 | Hurt |
| Select the lowest-confidence NATURE operator per utterance | Stack-Prop+BERT evaluation · ATIS, SNIPS, NLU-ED (Avg.) | E2E accuracy: 80.6 -> 28.9 | Hurt |
| Apply one random NATURE operator per utterance | Bi-RNN evaluation · ATIS, SNIPS, NLU-ED (Avg.) | E2E accuracy: 74.3 -> 50.3 | Hurt |
| Select the lowest-confidence NATURE operator per utterance | Bi-RNN evaluation · ATIS, SNIPS, NLU-ED (Avg.) | E2E accuracy: 74.3 -> 23.0 | Hurt |
| Add an end-of-sentence filler (ablation) | Stack-Prop+BERT · ATIS, SNIPS, NLU-ED (Avg.) | E2E accuracy: 80.6 -> 40.3 | Hurt |
| Replace an adjective with a BERT candidate (ablation) | Stack-Prop+BERT · ATIS, SNIPS, NLU-ED (Avg.) | E2E accuracy: 80.6 -> 44.7 | Hurt |
| Replace an adverb with a BERT candidate (ablation) | Stack-Prop+BERT · ATIS, SNIPS, NLU-ED (Avg.) | E2E accuracy: 80.6 -> 43.7 | Hurt |
| Augment training and validation data using NLPaug strategies | Stack-Prop+BERT · ATIS, SNIPS, NLU-ED (Avg.) | E2E accuracy on Hard: 28.9 -> 40.6 (+11.7) | Improved |
| Augment training and validation data using NLPaug strategies | Stack-Prop+BERT · ATIS, SNIPS, NLU-ED (Avg.) | E2E accuracy on Orig: 80.6 -> 78.3 (-2.3) | Hurt |
| Apply Checklist punctuation perturbations | Stack-Prop+BERT · ATIS, SNIPS, NLU-ED (Avg.) | E2E accuracy: 80.6 -> 26.6 | Hurt |
Closest papers by meaning
- Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversation (NeurIPS 2024 · similarity 0.86)
- EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge (NeurIPS 2025 · similarity 0.86)
- SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words (NeurIPS 2024 · similarity 0.85)
- $\tau$-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains (ICML 2026 · similarity 0.85)
- Aligning Spoken Dialogue Models from User Interactions (ICML 2025 · similarity 0.85)
- A Variational Framework for Improving Naturalness in Generative Spoken Language Models (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
