Research map / Large language models / Language, safety and interpretability

DialectLLM: A Dialect-Aware Dialog[ue] Generation Framework Beyond Standard American English

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Faithfulness of LLM explanations

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces DialectLLM, a linguist-validated pipeline that generates parallel multi-turn English dialect dialogs (9 dialects) across lexical, orthographic and morphosyntactic features, separating user-appropriate from model-appropriate grammar, and builds DialectLLM-Bench to test dialect identification and response completion.

Problem

LLMs fail to identify non-SAE English dialects and produce stereotyped responses, while existing dialect data generators rely on eWAVE morphosyntactic rules alone and ignore lexical/orthographic variation and what models should actually generate.

Approach

Decompose dialect transformation into lexical, orthographic and morphosyntactic dimensions with native-linguist ratings, and split morphosyntactic features into user-appropriate and model-appropriate sets, so that generated assistant responses reproduce far fewer dialect grammar features than user turns.

Key results

• Human native-speaker annotators preferred DialectLLM over Trans-EnV in about 99% of comparisons for user and model utterances.
• Frontier models such as Claude-Opus-4.1 score under 70% average accuracy and correctly classify Canadian English only 33% of the time.
• SFT of Qwen3-8B raised turn-1 accuracy from about 30% to roughly 90%, surpassing frontier models.

Surprising findings

• Up to 90% of eWAVE morphosyntactic features were judged by linguists as inappropriate for model-generated responses.
• Smaller models score higher on short, ambiguous dialogs, and their success there reflects label ambiguity and bias toward high-resource dialects rather than genuine capability.

Limitations

• Limited to transcribable, country-level English dialects; speech/accent variation not covered.
• Reliance on LLM-generated candidate mappings and eWAVE rules, with linguist validation for only a subset of dialects (3 annotators per dialect for some).

Experimental setup

Datasets: DialectLLM-OL DialectLLM-User DialectLLM-Model eWAVE

Benchmarks: DialectLLM-Bench

Metrics: accuracy human preference win rate

Baselines: Trans-EnV Multi-Value

Builds on: eWAVE Trans-EnV Multi-Value

Closest papers by meaning

• Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversation (NeurIPS 2024 · similarity 0.88)
• SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words (NeurIPS 2024 · similarity 0.87)
• A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators (AAAI 2024 · similarity 0.87)
• Aligning Spoken Dialogue Models from User Interactions (ICML 2025 · similarity 0.86)
• Improving Open-Domain Dialogue Response Generation with Multi-Source Multilingual Commonsense Knowledge (AAAI 2024 · similarity 0.86)
• Trans-EnV: A Framework for Evaluating the Linguistic Robustness of LLMs Against English Varieties (NeurIPS 2025 · similarity 0.86)