Research map / Large language models / Language, safety and interpretability
DSGram: Dynamic Weighting Sub-Metrics for Grammatical Error Correction in the Era of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DSGram, a reference-free GEC evaluation framework combining Semantic Coherence, Edit Level, and Fluency with LLM-generated AHP weights per sentence, plus two human/LLM-annotated datasets.
Problem
Reference-based GEC metrics (M2, ERRANT, GLEU) and existing reference-free metrics (SOME) fail to capture over-correction and fixed sub-metric weighting does not adapt to context when evaluating LLM-based GEC systems.
Approach
Replaces SOME's Grammaticality/Fluency/Meaning sub-metrics with Semantic Coherence, Edit Level (to penalize over-editing) and Fluency, and derives per-sentence criterion weights by having LLMs build AHP pairwise judgment matrices with consistency checks.
Key results
- DSGram (GPT-4 weighted) achieves system-level Pearson 0.927 on SEEDA-E and sentence-level accuracy 0.776 on SEEDA-S, exceeding SOME and IMPARA on several settings.
- GPT-4 zero-shot reaches Pearson 0.772 overall with DSGram-Eval human scores, while LLaMA3-70B zero-shot reaches 0.607 and fine-tuned LLaMA3-8B reaches 0.419.
- AHP-weighted human score correlates 0.8764 with human overall score versus 0.8544 for equal weighting; Cronbach's Alpha 0.76-0.82 across three text domains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GPT-4-generated dynamic weights instead of equal weighting (ablation) | DSGram · DSGram-Eval | Pearson correlation with human overall scores: 0.8544 -> 0.8764 | Helped |
| fine-tune LLaMA3-8B on DSGram-LLMs instead of using 5-shot prompting (ablation) | LLaMA3-8B · DSGram-Eval | Pearson correlation with human overall scores: 0.373 -> 0.419 | Helped |
| fine-tune LLaMA2-13B on DSGram-LLMs instead of using 5-shot prompting (ablation) | LLaMA2-13B · DSGram-Eval | Pearson correlation with human overall scores: 0.306 -> 0.382 | Helped |
| use LLaMA3-70B weighting instead of GPT-4 weighting (ablation) | DSGram · SEEDA-S system-level | Spearman ρ: 0.909 -> 0.853 | Hurt |
| use LLaMA3-70B weighting instead of GPT-4 weighting (ablation) | DSGram · SEEDA-E sentence-level | Accuracy: 0.750 -> 0.649 | Hurt |
| use equal weighting instead of GPT-4 dynamic weighting (ablation) | DSGram · SEEDA-E system-level | Pearson r: 0.922 -> 0.927 | Hurt |
| use GPT-4 zero-shot scoring rather than LLaMA3-70B zero-shot scoring | DSGram scoring · DSGram-Eval | Pearson correlation, Overall: 0.607 -> 0.772 | Helped |
| replace the three SOME sub-metrics with Semantic Coherence, Edit Level, and Fluency | DSGram · SEEDA | correlation with human rankings | Mixed |
Surprising findings
- Fine-tuned open LLaMA models (LLaMA2-13B, LLaMA3-8B) correlate poorly with humans (overall 0.382 and 0.419), below zero-shot LLaMA3-70B.
- REF-F has the highest SOME score but underperforms markedly under DSGram, showing the reference-free metrics disagree.
Limitations
- Open LLaMA models struggle to follow structured prompts and align poorly with human scores unless strong models like GPT-4 are used.
- Validation is limited to SEEDA and a manually annotated DSGram-Eval set; generalization across LLMs is only partly tested.
- Weight consistency is assessed by Cronbach's Alpha rather than against human weight labels across diverse domains.
Experimental setup
Datasets: SEEDA BEA-2019 CoNLL-2014 SOME dataset DSGram-LLMs DSGram-Eval OpenSubtitles Justia CaseLaw Wikipedia
Benchmarks: SEEDA DSGram-Eval
Metrics: Pearson correlation Spearman correlation Accuracy Kendall tau Cronbach's Alpha consistency ratio
Baselines: M2 ERRANT GLEU SOME IMPARA Scribendi Score Perplexity
Builds on: Analytic Hierarchy Process SOME IMPARA chain-of-thought prompting
Closest papers by meaning
- ScholarGEC: Enhancing Controllability of Large Language Model for Chinese Academic Grammatical Error Correction (AAAI 2025 · similarity 0.90)
- ALRMR-GEC: Adjusting Learning Rate Based on Memory Rate to Optimize the Edit Scorer for Grammatical Error Correction (AAAI 2025 · similarity 0.88)
- Neither Valid nor Reliable? Investigating the Use of LLMs as Judges (NeurIPS 2025 · similarity 0.86)
- Importance Weighting Can Help Large Language Models Self-Improve (AAAI 2025 · similarity 0.86)
- Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysis (ICLR 2024 · similarity 0.86)
- Tuning LLM Judge Design Decisions for 1/1000 of the Cost (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.