Research map / Large language models / Language, safety and interpretability
Improving Factual Error Correction by Learning to Inject Factual Errors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LIFE, a distantly supervised factual error correction method that trains a T5 corruptor to inject factual errors into correct claims via mask-then-corrupt, filters the synthetic pairs, and trains a T5 corrector end-to-end without a masker at test time.
Problem
Factual error correction (FEC) without paired false/correct claims; existing mask-then-correct distantly supervised methods depend on a masker that must accurately identify factual errors, which is hard without paired training data.
Approach
Instead of identifying errors before correcting, train a corruptor (masked-claim reconstruction conditioned on evidence) to inject factual errors into correct claims, generate synthetic paired data, filter it with a Levenshtein filter and a fact-verification-based filter, and train a T5 corrector that needs no masker at test time.
Key results
- LIFE reaches 65.59 SARI Final and 66.51 RG-2 on FECDATA test with retrieved evidence, beating the best distantly supervised baseline VENCE (55.0 with RoBERTa-large / 53.6 with BERT-base) by 10.59 points.
- LIFE outperforms 8-shot ChatGPT (58.43 SARI Final) by about 7.16 points.
- In human evaluation, LIFE achieves 58.7% corrected versus 42.0% for 8-shot ChatGPT, though it trails the fully supervised T5-base (86.7%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LIFE instead of VENCE (RoBERTa-large) | Distantly supervised FEC on FECDATA · FECDATA test, retrieved evidence | SARI Final: 55.0 -> 65.59 | Helped |
| use LIFE instead of 8-shot ChatGPT | FEC on FECDATA · FECDATA test, retrieved evidence | SARI Final: 58.43 -> 65.59 | Helped |
| replace retrieved evidence with gold evidence | LIFE · FECDATA test | SARI Final: 65.59 -> 70.72 | Helped |
| use heuristic rather than random masking during corruptor training (ablation) | LIFE · FECDATA test, retrieved evidence | SARI Final: 60.04 -> 65.59; 58.54 -> 61.30 | Helped |
| use random rather than heuristic masking during corruptor inference (ablation) | LIFE · FECDATA test, retrieved evidence | SARI Final: 58.54 -> 65.59; 61.30 -> 65.59 | Helped |
| use subword-level rather than word-level masking (ablation) | LIFE · FECDATA test, retrieved evidence | SARI Final: 65.59 -> 65.18 | No clear effect |
| merge consecutive word-level mask tokens (ablation) | LIFE · FECDATA test, retrieved evidence | SARI Final: 65.59 -> 64.82 | Hurt |
| remove the Levenshtein filter (LF) (ablation) | LIFE · FECDATA test, retrieved evidence | SARI Final: 65.59 -> 64.76 | Hurt |
| remove the fact verification classifier-based filter (FVCF) (ablation) | LIFE · FECDATA test, retrieved evidence | SARI Final: 65.59 -> 65.07 | Hurt |
| remove both LF and FVCF (ablation) | LIFE · FECDATA test, retrieved evidence | SARI Final: 65.59 -> 64.12 | Hurt |
Surprising findings
- Heuristic masking during training combined with random masking during testing works best, reversing the intuition that the same strategy should be used in both phases.
- The 8-shot T5-base baseline does not outperform distantly supervised VENCE, and LLM few-shot prompting performs much better than fine-tuning T5 on 8 examples.
Limitations
- Corruptor training depends on the masker correctly identifying factual errors; when it fails, the corruptor does not learn to inject errors.
- A significant gap remains between LIFE and the fully supervised T5-base, possibly due to noise in synthetic data.
- Performance with retrieved evidence is much lower than with gold evidence, indicating retrieval is a bottleneck.
Experimental setup
Datasets: FECDATA FEVER
Benchmarks: FECDATA
Metrics: SARI Keep SARI Delete SARI Add SARI Final ROUGE-2 human grammaticality/support/correction
Baselines: Fully supervised T5-base DEPN T5MC T5MC-MLM T5MC-V VENCE 8-shot T5-base 8-shot text-ada-001 8-shot text-babbage-001 8-shot ChatGPT
Builds on: T5 VENCE FEVER GENRE DPR
Closest papers by meaning
- Fine-Tuning Language Models for Factuality (ICLR 2024 · similarity 0.85)
- Unmasking and Improving Data Credibility: A Study with Datasets for Training Harmless Language Models (ICLR 2024 · similarity 0.84)
- FactTest: Factuality Testing in Large Language Models with Finite-Sample and Distribution-Free Guarantees (ICML 2025 · similarity 0.84)
- Is Factuality Enhancement a Free Lunch For LLMs? Better Factuality Can Lead to Worse Context-Faithfulness (ICLR 2025 · similarity 0.84)
- Unsupervised Pretraining for Fact Verification by Language Model Distillation (ICLR 2024 · similarity 0.84)
- Mask-DPO: Generalizable Fine-grained Factuality Alignment of LLMs (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.