Research map / Architectures and efficiency / Attention and transformers
Decoupling The "What" and "Where" With Polar Coordinate Positional Embedding
Anand Gopalakrishnan, Róbert Csordás, Jürgen Schmidhuber, Michael Curtis Mozer
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace RoPE with PoPE | Transformer · Indirect Indexing | accuracy: 11.16 -> 94.82 | Improved |
| replace RoPE with PoPE | Transformer · JSB | NLL: 0.5081 -> 0.4889 | Improved |
| replace RoPE with PoPE | Transformer · MAESTRO | NLL: 1.501 -> 1.486 | Improved |
| replace RoPE with PoPE | Transformer · Human Reference Genome | NLL: 4.217 -> 4.152 | Improved |
| replace RoPE with PoPE | Transformer 774M · OpenWebText val | perplexity: 15.85 -> 15.45 | Improved |
| replace RoPE with PoPE | Transformer 124M · OpenWebText val | perplexity: 21.55 -> 21.33 | Improved |
| add softplus activation sigma (ablation) | PoPE 124M · OpenWebText val | perplexity: 21.57 -> 21.33 | Improved |
| use softplus instead of ReLU for sigma (ablation) | PoPE 124M · OpenWebText val | perplexity: 21.55 -> 21.33 | Improved |
| add learnable bias delta (ablation) | PoPE 124M · OpenWebText val | perplexity: 21.42 -> 21.33 | Improved |
| replace RoPE with PoPE | Transformer 124M · Zero-shot downstream average | accuracy: 45.33 -> 46.19 | Improved |
| replace RoPE with PoPE | Transformer 253M · Zero-shot downstream average | accuracy: 48.76 -> 48.78 | No clear effect |
| replace RoPE with PoPE | Transformer 124M · LAMBADA zero-shot | accuracy: 28.74 -> 27.67 | Hurt |
Closest papers by meaning
- Round and Round We Go! What makes Rotary Positional Encodings useful? (ICLR 2025 · similarity 0.89)
- Rethinking Addressing in Language Models via Contextualized Equivariant Positional Encoding (ICML 2025 · similarity 0.89)
- What Rotary Position Embedding Can Tell Us: Identifying Query and Key Weights Corresponding to Basic Syntactic or High-level Semantic Information (NeurIPS 2024 · similarity 0.89)
- PaTH Attention: Position Encoding via Accumulating Householder Transformations (NeurIPS 2025 · similarity 0.88)
- Base of RoPE Bounds Context Length (NeurIPS 2024 · similarity 0.88)
- Wavelet-based Positional Representation for Long Context (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
