Research map / Architectures and efficiency / Attention and transformers
Attention Biasing and Context Augmentation for Zero-Shot Control of Encoder-Decoder Transformers for Natural Language Generation
Devamanyu Hazarika, Mahdi Namazifar, Dilek Hakkani-Tür
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Bias cross-attention toward the knowledge snippet | BART-base response generation · Topical-Chat | informativeness (human evaluation): 3.43 -> 3.84 | Improved |
| Bias cross-attention toward the knowledge snippet | BART-base response generation · Topical-Chat rare test set | ROUGEk: 0.28 -> 0.49 | Improved |
| Bias cross-attention toward the knowledge snippet | BART-base response generation · Topical-Chat | fluency (human evaluation) | No clear effect |
| Bias cross-attention toward dialog history instead of knowledge (ablation) | BART-base response generation · Topical-Chat rare test set | ROUGEk: 0.28 -> 0.20 | Hurt |
| Gradually shift cross-attention bias from dialog history toward knowledge (ablation) | BART-base response generation · Topical-Chat | informativeness (human evaluation): 3.43 -> 3.80 | Improved |
| Increase knowledge bias from 1 to 50 while keeping dialog bias at 1 (ablation) | BART-base response generation · Topical-Chat frequent test set | PPLr: 9.66 -> 12.23 | Hurt |
| Add a question control code (ablation) | BART-base question generation · Topical-Chat | percentage of responses containing questions: 29.1% -> 29.3% (↑0.2%) | No clear effect |
| Add question control code and attention bias toward it during early decoding | BART-base question generation · Topical-Chat | percentage of responses containing questions: 29.1% -> 35.2% (↑6.1%) | Improved |
| Add feedback control code and attention bias toward it during early decoding | BART-base response generation · Topical-Chat | feedback responses (%): 17.7 -> 22.1 | Improved |
| Add positive-sentiment control code without attention bias (ablation) | BART-base response generation · Topical-Chat | p(positive|y): 0.561±0.02 -> 0.551±0.01 | No clear effect |
| Add positive-sentiment control code and attention bias toward it during early decoding | BART-base response generation · Topical-Chat | p(positive|y): 0.561±0.02 -> 0.619±0.03 | Improved |
Closest papers by meaning
- Generating Training Data with Language Models: Towards Zero-Shot Language Understanding (NeurIPS 2022 · similarity 0.89)
- What Language Model Architecture and Pretraining Objective Works Best for Zero-Shot Generalization? (ICML 2022 · similarity 0.89)
- Evaluating the Zero-shot Robustness of Instruction-tuned Language Models (ICLR 2024 · similarity 0.85)
- Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models (NeurIPS 2024 · similarity 0.85)
- Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Models (NeurIPS 2024 · similarity 0.85)
- Gen-Z: Generative Zero-Shot Text Classification with Contextualized Label Descriptions (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
