Research map / Large language models / Language, safety and interpretability
Quantifying the Plausibility of Context Reliance in Neural Machine Translation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PECoRe, a two-step interpretability framework that identifies context-sensitive generated tokens (via contrastive metrics) and links them to contextual cues (via contrastive attribution), and uses it to measure plausibility of context reliance in context-aware machine translation.
Problem
Existing plausibility evaluations of context usage in language models only check single hand-picked generation steps on a few artificial benchmarks, without identifying which generated tokens are context-dependent and which context cues drive them.
Approach
A unified end-to-end framework: contrastive context-sensitive token identification (CTI) using context vs. no-context next-token distributions, followed by contrastive contextual cue imputation (CCI) via attribution, yielding cue-target pairs that are compared against human rationales and applied to unannotated translations.
Key results
- KL-Divergence performs on par or better than pointwise metrics (LR, P-CXMI) for context-sensitive token identification, while context saliency performs poorly.
- Gradient-based attribution (nabla_KL and nabla_diff) is the most robust CCI choice across SCAT+ and DiscEval-MT, while Attention Best suffers most in the end-to-end setting.
- Context-aware fine-tuning raises OpusMT Large S+T_ctx on SCAT+ from 29.0 to 40.3 BLEU and disambiguation ok from 0.16 to 0.83.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune OpusMT Small with source and target context | OpusMT Small · SCAT+ | BLEU: 29.1 -> 39.1 | Helped |
| Fine-tune OpusMT Large with source and target context | OpusMT Large · SCAT+ | ok: 0.16 -> 0.83 | Helped |
| Fine-tune mBART-50 with source and target context | mBART-50 · DiscEval-MT anaphora | BLEU: 33.4 -> 49.0 | Helped |
| Fine-tune OpusMT Small with source and target context | OpusMT Small · DiscEval-MT lexical choice | ok: 0.29 -> 0.36 | Helped |
| Use source-only context instead of source and target context | Context-aware MT models · En→Fr test sets | BLEU | Mixed |
Surprising findings
- Attention Best, the best-case attention head, suffers the most when moving from gold CTI tags to end-to-end application, indicating attention generalizes poorly as an explanation.
- Source-only context models yield more easily detected contextual cues than source+target context models across all attribution methods.
Limitations
- Plausibility evaluation limited to two discourse phenomena (anaphora and lexical choice) and En-Fr translation for the annotated sets.
- Gold context-sensitive tags are obtained by aligning reference annotations to model generations, which may be inaccurate.
- Per-example variability in attribution plausibility remains high.
Experimental setup
Datasets: IWSLT 2017 English-French SCAT DiscEval-MT Flores-101
Benchmarks: SCAT+ DiscEval-MT (ana) DiscEval-MT (lex) Flores-101
Metrics: Macro F1 AUPRC BLEU COMET ok ok-cs
Baselines: Attention Mean Attention Best Relative context saliency Likelihood Ratio P-CXMI stratified random baseline
Builds on: contrastive conditioning contrastive gradient norm attribution likelihood ratio P-CXMI SCAT
Closest papers by meaning
- Context Steering: Controllable Personalization at Inference Time (ICLR 2025 · similarity 0.87)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.86)
- AttriBoT: A Bag of Tricks for Efficiently Approximating Leave-One-Out Context Attribution (ICLR 2025 · similarity 0.86)
- Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass (ICLR 2025 · similarity 0.86)
- What is Wrong with Perplexity for Long-context Language Modeling? (ICLR 2025 · similarity 0.86)
- Forking Paths in Neural Text Generation (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.