Research map / Architectures and efficiency / Attention and transformers
Quantum Doubly Stochastic Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Replaces the softmax in ViT self-attention with a parametric variational quantum circuit (QontOT) that emits doubly stochastic matrices, plus a QR-based quantum-inspired variant, and evaluates on small image classification tasks.
Problem
Softmax attention is right-stochastic only, and doubly stochastic attention via Sinkhorn's algorithm is iterative, approximate, non-parametric and can be unstable; ViTs also train unstably on small-scale data.
Approach
Extends the QontOT variational quantum circuit (which outputs doubly stochastic matrices via U ⊙ Ū for unitary U) from scalar to matrix data injection so it can replace softmax inside self-attention, yielding a parametric doubly stochastic Transformer (QDSFormer); also proposes a QR-decomposition-based quantum-inspired DSM operator.
Key results
- On FashionMNIST and MNIST, QontOT-based QDSFormer with 2-4 ViT layers exceeded softmax ViT and Sinkhorn attention; e.g., 1-layer MNIST: QontOT 93.9 vs Softmax 89.1 (with high variance for Softmax).
- On MedMNIST (2-layer ViT), QontOT achieved mean accuracy 74.3% vs 73.0% for Softmax, with the best result on 5 of 7 datasets reported as significant.
- On the Eureka compositional task, QDSFormer learned within 100 epochs, reportedly giving ~30% accuracy improvement over a standard ViT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace softmax attention with QontOT | 2-layer ViT · MedMNIST mean test | accuracy: 73.0 -> 74.3 | Helped |
| Replace softmax attention with QontOT | 2-layer ViT · MedMNIST test datasets | accuracy | Mixed |
| Replace softmax attention with QontOT | 1-layer ViT · IR spectra, 8x8 attention | Micro-F1: 81.68_{\pm 0.18} -> 81.70_{\pm 0.05} | No clear effect |
| Replace softmax attention with QontOT | 1-layer ViT · IR spectra, 16x16 attention | Micro-F1: 81.55_{\pm 0.07} -> 81.06_{\pm 0.27} | Hurt |
| Increase QontOT circuit depth from 1 to 32 layers (ablation) | QDSFormer · MNIST validation | accuracy: 83.4_{\pm 0.73} -> 94.2_{\pm 0.30} | Helped |
| Increase QontOT circuit depth from 1 to 16 layers (ablation) | QDSFormer · FashionMNIST validation | accuracy: 87.7_{\pm 0.22} -> 90.0_{\pm 0.15} | Helped |
| Increase QontOT circuit depth from 16 to 32 layers (ablation) | QDSFormer · FashionMNIST validation | accuracy: 90.0_{\pm 0.15} -> 90.0_{\pm 0.13} | No clear effect |
| Use the Simple ansatz instead of Parted (ablation) | 2-layer QDSFormer · FashionMNIST, 1 circuit layer | accuracy: 87.7_{\pm 0.22} -> 88.0_{\pm 0.10} | No clear effect |
| Use the Simple ansatz instead of Trotter (ablation) | 2-layer QDSFormer · FashionMNIST, 8 circuit layers | accuracy: 88.4_{\pm 0.20} -> 89.9_{\pm 0.15} | Helped |
| Jointly optimize circuit and Transformer parameters rather than use a static circuit (ablation) | QDSFormer · MNIST and FashionMNIST | accuracy | Hurt |
| Optimize the circuit more frequently during mixed training (ablation) | QDSFormer · MNIST | accuracy | No clear effect |
| Replace the Simple ansatz with other tested circuit ansatz types (ablation) | QDSFormer · FashionMNIST | accuracy | Mixed |
Surprising findings
- The static (non-trainable, fixed-parameter) QontOT attention performed as well as or better than the jointly optimized end-to-end version.
- Sinkhorn with 3 or 21 iterations did not converge to a doubly stochastic matrix on trained Sinkformer attention matrices, and QR decomposition produced fewer unique DSMs than QontOT.
Limitations
- All experiments use exact statevector simulation rather than real quantum hardware; hardware noise pushes DSMs toward uniform (high entropy).
- Experiments limited to small-scale datasets and small ViTs (hidden dim 128, one head), and no hyperparameter optimization was performed.
- End-to-end differentiable training of the circuit was slower and less accurate, possibly due to barren plateaus; QR attention scales cubically in sequence length.
Experimental setup
Datasets: MNIST Fashion-MNIST MedMNIST (OCTMNIST, PneumoniaMNIST, TissueMNIST, OrganAMNIST, OrganCMNIST, OrganSMNIST, BreastMNIST) Compositional Eureka dataset IR spectra of molecules
Benchmarks: MNIST Fashion-MNIST MedMNIST Compositional Eureka dataset IR spectra dataset
Metrics: validation accuracy test accuracy micro-F1 Frobenius distance Shannon entropy number of unique DSMs
Baselines: Softmax ViT Sinkformer (Sinkhorn) Softmax_sigma^2 (NormSoftmax) QR decomposition attention
Builds on: Sinkformer QontOT Sinkhorn's algorithm NormSoftmax
Closest papers by meaning
- The Quantization Benefits of Residual-Free Transformers (NeurIPS 2026 · similarity 0.83)
- Winning Lottery Tickets in Neural Networks via a Quantum-Inspired Classical Algorithm (NeurIPS 2026 · similarity 0.83)
- Transformer-VQ: Linear-Time Transformers via Vector Quantization (ICLR 2024 · similarity 0.83)
- On the Relation between Trainability and Dequantization of Variational Quantum Learning Models (ICLR 2025 · similarity 0.83)
- Characterizing Trainability of Instantaneous Quantum Polynomial Circuit Born Machine (NeurIPS 2026 · similarity 0.82)
- FIMA-Q: Post-Training Quantization for Vision Transformers by Fisher Information Matrix Approximation (CVPR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.