Research map / Architectures and efficiency / Attention and transformers
Time-aware Large Kernel Convolutions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TaLK Convolution instead of the strongest listed baseline | Machine translation · WMT En-De newstest2014 | BLEU: 29.7 -> 29.6 | No clear effect |
| Use TaLK Convolution instead of the strongest listed baseline | Machine translation · WMT En-Fr newstest2014 | BLEU: 43.2 -> 43.2 | No clear effect |
| Use TaLK Convolution instead of the strongest listed baseline | Machine translation · IWSLT De-En | BLEU: 35.2 -> 35.5 | Helped |
| Use TaLK Convolution instead of Dynamic Convolution | Abstractive summarization · CNN-DailyMail | Rouge-1: 39.84 -> 40.59 | Helped |
| Use TaLK Convolution instead of Dynamic Convolution | Abstractive summarization · CNN-DailyMail | Rouge-2: 16.25 -> 18.97 | Helped |
| Use TaLK Convolution instead of Dynamic Convolution | Abstractive summarization · CNN-DailyMail | Rouge-L: 36.73 -> 36.81 | Helped |
| Replace Dynamic Convolutions with TaLK Convolution | Language modeling · WikiText-103 test | perplexity: 25.0 -> 23.3 | Helped |
| Use TaLK Convolution rather than Self-Attention | Sequence encoding · Encoding, n=10,000 | iter/sec: OOM -> 92 | Helped |
| Add output normalization to summation kernels (ablation) | TaLK Convolution · IWSLT De-En validation | BLEU: diverges -> 35.70 ± 0.1 | Helped |
| Increase maximum offsets to 1,3,7,15x4 (ablation) | TaLK Convolution with output normalization · IWSLT De-En validation | BLEU: 35.70 ± 0.1 -> 36.23 ± 0.1 | Helped |
| Add dropout to predicted offsets with p=0.1 (ablation) | TaLK Convolution with increased maximum offsets · IWSLT De-En validation | BLEU: 36.23 ± 0.1 -> 36.37 ± 0.05 | Helped |
| Increase head count to 512 (ablation) | TaLK Convolution with offset dropout · IWSLT De-En validation | BLEU: 36.37 ± 0.05 -> 36.51 ± 0.07 | Helped |
Closest papers by meaning
- Curse of High Dimensionality Issue in Transformer for Long Context Modeling (ICML 2025 · similarity 0.87)
- LoCoCo: Dropping In Convolutions for Long Context Compression (ICML 2024 · similarity 0.86)
- Reparameterized Multi-Resolution Convolutions for Long Sequence Modelling (NeurIPS 2024 · similarity 0.86)
- Tensor Product Attention Is All You Need (NeurIPS 2025 · similarity 0.86)
- A Training-Free Sub-quadratic Cost Transformer Model Serving Framework with Hierarchically Pruned Attention (ICLR 2025 · similarity 0.86)
- Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsity (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.