Research map / Architectures and efficiency / Attention and transformers
Enhancing Transformers Through Conditioned Embedded Tokens
Hemanth Saratchandran, Simon Lucey
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add conditioned embedded tokens | ViT-B · ImageNet-1k (from scratch) | top-1 accuracy: 80.3 -> 81.3 | Improved |
| add conditioned embedded tokens | Swin-B · ImageNet-1k (from scratch) | top-1 accuracy: 83.1 -> 83.9 | Improved |
| add conditioned embedded tokens | DaViT-B · ImageNet-1k (from scratch) | top-1 accuracy: 83.6 -> 84.6 | Improved |
| add conditioned embedded tokens | XCiT-S · COCO Mask R-CNN | AP box: 44.9 -> 45.7 | Improved |
| add conditioned embedded tokens | XCiT-M · COCO Mask R-CNN | AP mask: 40.8 -> 41.4 | Improved |
| add conditioned embedded tokens | Crammed BERT · GLUE | CoLA accuracy: 48.9 -> 53.7 | Improved |
| add conditioned embedded tokens | Crammed BERT · GLUE | average: 78.6 -> 79.7 | Improved |
| add conditioned embedded tokens | GPT-2 · TinyStories | validation loss: 2.41 -> 2.36 | Improved |
| add conditioned embedded tokens | Nystromformer · LRA Retrieval | accuracy: 79.8 -> 80.9 | Improved |
| add conditioned embedded tokens | Nystromformer · LRA Image | accuracy: 39.9 -> 40.1 | Improved |
Closest papers by meaning
- Conditioned Initialization for Attention (ICLR 2026 · similarity 0.89)
- Spectral Conditioning of Attention Improves Transformer Performance (NeurIPS 2025 · similarity 0.88)
- Token Selection is a Simple Booster for Vision Transformers (TPAMI 2022 · similarity 0.88)
- Transformers Learn Faster with Semantic Focus (NeurIPS 2025 · similarity 0.87)
- Non-asymptotic Convergence of Training Transformers for Next-token Prediction (NeurIPS 2024 · similarity 0.87)
- In-context Convergence of Transformers (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
