Research map / Architectures and efficiency / Attention and transformers
Spline-based Transformers
Prashanth Chandran, Agon Serifi, Markus Gross, Moritz Bächer
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace positional encoding (ALiBi) with latent spline trajectory | transformer autoencoder · CIFAR-10 (32D) | MSE: 0.266 -> 0.107 | Improved |
| replace positional encoding with latent spline | transformer autoencoder · HumanML3D motion (64D) | MSE joint angle (deg): 0.059 -> 0.006 | Improved |
| replace positional encoding with latent spline on a high-dimensional latent | transformer autoencoder · Synthetic Bezier curves (64D) | reconstruction MSE: 3.88e-3 -> 3.87e-3 | No clear effect |
| replace positional encoding with latent spline on 3D face animation with larger latent | transformer autoencoder · 3D face animation (128D, 256D) | MSE: 128D: 1.48 -> 1.47; 32D: 1.58 -> 1.43 | Mixed |
| concatenate sinusoidal position to token embedding (ALiBi-Cat) instead of adding (ALiBi) (ablation) | ALiBi baseline · CIFAR-10 and AFHQ | MSE: CIFAR-10 128D: 0.107 -> 0.108 | No clear effect |
| use a large learning rate for the spline transformer (ablation) | Spline-based Transformer · AFHQ | validation loss | Hurt |
| use a too small learning rate for the spline transformer (ablation) | Spline-based Transformer · AFHQ | validation loss | Hurt |
| replace positional encoding with latent spline | transformer autoencoder · AFHQ | training convergence speed | Improved |
Closest papers by meaning
- Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding (NeurIPS 2025 · similarity 0.86)
- Algebraic Positional Encodings (NeurIPS 2024 · similarity 0.86)
- Predictive Geometry of Hidden Trajectories in Transformers (NeurIPS 2026 · similarity 0.85)
- Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling (NeurIPS 2024 · similarity 0.84)
- Functional Interpolation for Relative Positions improves Long Context Transformers (ICLR 2024 · similarity 0.84)
- Continuous Piecewise-Affine Based Motion Model for Image Animation (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
