Research map / Large language models / Language, safety and interpretability
Trans-EnV: A Framework for Evaluating the Linguistic Robustness of LLMs Against English Varieties
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Trans-EnV, an expert-guided, LLM-based framework that transforms Standard American English benchmarks into 38 dialect and ESL English varieties and evaluates seven LLMs on their robustness.
Problem
LLMs are mostly evaluated on Standard American English, and their performance may degrade on non-standard English varieties (regional dialects and ESL English), raising fairness concerns. Existing variety datasets are small, narrow, or hard to scale.
Approach
Combine expert-curated linguistic features (eWAVE dialect features, CEFR-based EGP features, and L1-specific ESL error features from learner corpora) with LLM-generated step-by-step transformation guidelines, applied sequentially by an LLM transformer and filtered by an LLM semantic checker to automatically convert SAE benchmark items into target varieties.
Key results
- Accuracy drops of up to 46.3% on ESL English varieties and up to 12.5% on dialect varieties across seven LLMs.
- Human evaluation found 88% of transformed samples valid overall (132/150), with most transformers above 90% except LLaMA-3.1-8B at 56%.
- Semantic checker LLaMA-3.3-70B reached 83.6% precision, 97.0% recall and 89.8% F1 against human labels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate LLMs on dialect varieties rather than SAE | LLMs · Non-standard English varieties | accuracy: up to 12.5% | Hurt |
| Evaluate LLMs on ESL English varieties rather than SAE | LLMs · Non-standard English varieties | accuracy: up to 46.3% | Hurt |
| Evaluate on dialect varieties rather than SAE | LLMs · Knowledge-based QA datasets | performance: 1.3% | Hurt |
| Evaluate on ESL English varieties rather than SAE | LLMs · Knowledge-based QA datasets | performance: 10.9% | Hurt |
| Evaluate on dialect varieties rather than SAE | LLMs · Reasoning-based QA datasets | performance: 4.7% | Hurt |
| Evaluate on ESL English varieties rather than SAE | LLMs · Reasoning-based QA datasets | performance: 22.6% | Hurt |
| Use CEFR A varieties instead of CEFR B varieties | LLMs · 4 of 6 QA datasets | mean score | Hurt |
| Use longer inputs | LLMs on five datasets other than GSM8K · MMLU, ARC, TruthfulQA, HellaSwag, WinoGrande | performance | Hurt |
| Use dialects linguistically farther from SAE | LLMs · ARC and WinoGrande | performance degradation | Hurt |
| Use ESL varieties from L1 categories 3 and 4 rather than categories 1 and 2 | LLMs · GSM8K and ARC | performance drop | Hurt |
| Use transformed variety instructions instead of original SAE instructions | LLMs on open-ended tasks · IFEval, AlpacaFarm, MT-Bench | performance | Hurt |
| Use a reasoning-specialized model | LLMs · Non-standard English varieties | robustness | Helped |
Surprising findings
- Reasoning-specialized models such as DeepSeek-R1-Distill-Llama-70B and o4-mini were more robust, and CEFR A varieties sometimes scored lower than B varieties as expected but not uniformly.
Limitations
- Focuses only on English varieties; extension to other languages needs language-specific resources.
- Evaluation limited to QA tasks, with open-ended tasks only as a preliminary experiment.
- Transformation quality depends on the LLM transformer; LLaMA-3.1-8B produced only 56% valid transformations.
Experimental setup
Datasets: MMLU ARC TruthfulQA GSM8K HellaSwag WinoGrande eWAVE English Grammar Profile CLC-FCE ICLE EFCamDat Oxford 5000
Benchmarks: MMLU ARC TruthfulQA GSM8K HellaSwag WinoGrande IFEval AlpacaFarm MT-Bench
Metrics: accuracy win rate instruction adherence accuracy MT-Bench score human validity rate precision recall F1
Baselines: Gemini 2.0 Flash GPT-4o-mini o4-mini Gemini 2.5 Pro LLaMA-3.3-70B-Instruct
Builds on: eWAVE English Grammar Profile CEFR
Closest papers by meaning
- Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages (AAAI 2025 · similarity 0.88)
- Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.88)
- Bridging the Gap: Enhancing LLM Performance for Low-Resource African Languages with New Benchmarks, Fine-Tuning, and Cultural Adjustments (AAAI 2025 · similarity 0.87)
- Neither Valid nor Reliable? Investigating the Use of LLMs as Judges (NeurIPS 2025 · similarity 0.87)
- Measuring what Matters: Construct Validity in Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.87)
- Paloma: A Benchmark for Evaluating Language Model Fit (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.