Research map / Architectures and efficiency / Attention and transformers
CAR-Transformer: Cross-Attention Reinforcement Transformer for Cross-Lingual Summarization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAR, a parameter-free auxiliary objective that treats cross-attention awareness as a policy over source tokens to form pseudo-monolingual summaries and trains it with self-critical policy gradient, improving mBART50 on cross-lingual summarization.
Problem
Cross-lingual summarization typically relies on costly multi-task training or knowledge distillation with monolingual summarization to improve performance, and the reasons for these gains are poorly explained.
Approach
Formulate a pseudo-summary policy parameterized by the decoder's cross-attention awareness vector (average attention per source token), sample pseudo-monolingual summaries, reward them via ROUGE against the ground-truth monolingual summary, and optimize with self-critical policy gradient, adding no extra parameters and disabling the module at inference.
Key results
- On WikiLingua ko-en, CAR reaches 26.03 ROUGE-L vs 22.36 for vanilla mBART (+3.67) and 24.48 for the best multi-task baseline (2-step), i.e. +1.48 over the best multi-task approach by the abstract's account.
- CAR significantly outperforms mBART on most WikiLingua, GlobalVoice (ar-en, fr-en) tasks, but not on CrossSum significance tests.
- Training costs: CAR adds about 1.3x total time versus mBART, while 2-step requires 2x steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add CAR cross-attention reinforcement | mBART50-large fine-tuning · WikiLingua Korean-to-English | ROUGE-L F1: 24.48 -> 26.03 | Helped |
| add CAR cross-attention reinforcement | mBART50-large fine-tuning · WikiLingua Hindi-to-English | ROUGE-L F1: 25.17 -> 25.32 | Helped |
| add CAR cross-attention reinforcement | mBART50-large fine-tuning · WikiLingua Czech-to-English | ROUGE-L F1: 21.96 -> 22.46 | Helped |
| add CAR cross-attention reinforcement | mBART50-large fine-tuning · WikiLingua Chinese-to-English | ROUGE-L F1: 24.83 -> 25.31 | Helped |
| add CAR cross-attention reinforcement | mBART50-large fine-tuning · WikiLingua Thai-to-English | ROUGE-L F1: 26.85 -> 27.00 | Helped |
| add CAR cross-attention reinforcement | mBART50-large fine-tuning · WikiLingua Turkish-to-English | ROUGE-L F1: 28.21 -> 29.11 | Helped |
| add CAR cross-attention reinforcement | mBART50-large fine-tuning · GlobalVoice Arabic-to-English | ROUGE-L F1: 19.00 -> 19.10 | Helped |
| add CAR cross-attention reinforcement | mBART50-large fine-tuning · GlobalVoice French-to-English | ROUGE-L F1: 22.31 -> 22.68 | Helped |
| add CAR cross-attention reinforcement | mBART50-large fine-tuning · CrossSum Arabic-to-English | ROUGE-L F1: 21.35 -> 21.37 | Helped |
| add CAR cross-attention reinforcement | mBART50-large fine-tuning · CrossSum French-to-English | ROUGE-L F1: 22.45 -> 22.54 | Helped |
| add CLS-MS multi-task training to CAR (ablation) | CAR · GlobalVoice Arabic-to-English | ROUGE-2: 6.41 -> 5.00 | Hurt |
| add CLS-MS multi-task training to CAR (ablation) | CAR · GlobalVoice Arabic-to-English | ROUGE-L: 19.10 -> 18.12 | Hurt |
Surprising findings
- Incorporating CLS-MS as a hybrid with CAR harms performance on GlobalVoice (ar-en ROUGE-1 drops to 23.34 from 25.42).
Limitations
- Reward computation (ROUGE) runs on CPU and increases training wall-clock time.
- No significant improvement over mBART on the CrossSum dataset.
- Relies on ground-truth monolingual summaries, which must be available or generated by an mT5 model for GlobalVoice and CrossSum.
Experimental setup
Datasets: WikiLingua GlobalVoice CrossSum XLSum
Benchmarks: WikiLingua GlobalVoice CrossSum
Metrics: ROUGE-1 ROUGE-2 ROUGE-L F1 GSB preference Key Awareness Ratio
Baselines: mBART CLS-MS CLS-MT 2-step attention relay
Builds on: mBART self-critical sequence training attention relay REINFORCE
Closest papers by meaning
- Improving Neural Cross-Lingual Abstractive Summarization via Employing Optimal Transport Distance for Knowledge Distillation (AAAI 2022 · similarity 0.89)
- MultiSumm: Towards a Unified Model for Multi-Lingual Abstractive Summarization (AAAI 2020 · similarity 0.87)
- MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks (ICLR 2026 · similarity 0.85)
- Distilling Structured Rationale from Large Language Models to Small Language Models for Abstractive Summarization (AAAI 2025 · similarity 0.83)
- Order-Level Attention Similarity Across Language Models: A Latent Commonality (NeurIPS 2025 · similarity 0.83)
- Soft Token Alignment for Cross-Lingual Reasoning (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.