Research map / Architectures and efficiency / Attention and transformers
The Inhibitor: ReLU and Addition-Based Attention for Efficient Transformers (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Inhibitor, an attention mechanism using Manhattan-distance scores and ReLU subtraction instead of dot-product and Softmax, aimed at efficient quantized and homomorphically encrypted Transformers.
Problem
Dot-product attention with Softmax requires variable-to-variable multiplication and Softmax evaluation, which are costly for quantized inference and especially hard under homomorphic encryption.
Approach
Replace query-key dot products with an L1 (Manhattan) distance score, shift it by a threshold, and aggregate values by subtracting the scores inside a ReLU, so attention needs only addition and ReLU with no Softmax or value multiplication.
Key results
- Inhibitor attention matches dot-product attention on four tasks: adding MSE 0.12% vs 0.11%, MNIST 97.9% vs 98.2%, IMDB 87.3% vs 87.2%, IAMW edit distance 18.1 vs 17.9.
- Plaintext integer 16-bit CPU timing is 30%-50% faster than dot-product attention at sequence lengths 32-256 (e.g., 63.1 µs vs 98.6 µs at length 32).
- Under TFHE encryption, execution time is roughly 3-6x faster at sequence lengths 2-16 (e.g., 0.749 s vs 2.68 s at length 2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace dot-product and Softmax attention with Inhibitor attention | Transformer · Adding | MSE: 0.11% -> 0.12% | No clear effect |
| Replace dot-product and Softmax attention with Inhibitor attention | Transformer · MNIST | accuracy: 98.2% -> 97.9% | No clear effect |
| Replace dot-product and Softmax attention with Inhibitor attention | Transformer · IMDB | accuracy: 87.2% -> 87.3% | No clear effect |
| Replace dot-product and Softmax attention with Inhibitor attention | Transformer · IAMW | edit distance: 17.9 -> 18.1 | No clear effect |
| Use Inhibitor attention instead of dot-product attention | Integer 16-bit plaintext CPU implementation · Sequence lengths 32, 64, 128, and 256 | execution time: 98.6 µs -> 63.1 µs; 330 µs -> 178 µs; 1.2 ms -> 577 µs; 4.48 ms -> 2.5 ms | Helped |
| Use Inhibitor attention instead of dot-product attention | TFHE encrypted CPU implementation · Sequence lengths 2, 4, 8, and 16 | execution time: 2.68 s -> 0.749 s; 22.4 s -> 8.56 s; 107 s -> 23.8 s; 828 s -> 127 s | Helped |
Surprising findings
- Inhibitor attention is slower than dot-product attention at length 256 in plaintext (2.5 ms vs 4.48 ms in the reverse direction of savings is not the case; Inhibitor is faster there), while under encryption it is slower at length 128 and 16 (127 s vs 828 s at 16 is faster, so the trend is mixed).
Limitations
- Experiments use one-layer, small Transformers without hyperparameter tuning and are not state-of-the-art.
- Only simple benchmarks tested; no large-scale pretraining or modern NLP benchmarks.
- Encrypted timing uses small embedding dimension (2) and up to 8-bit precision.
Experimental setup
Datasets: adding problem MNIST IMDB IAM Handwriting Database
Benchmarks: adding problem MNIST IMDB IAM Handwriting Database
Metrics: mse accuracy edit distance execution time
Baselines: dot-product attention Transformer
Builds on: Transformer ReLUformer Fastformer CTC loss
Closest papers by meaning
- Converting Transformers to Polynomial Form for Secure Inference Over Homomorphic Encryption (ICML 2024 · similarity 0.86)
- Encryption-Friendly LLM Architecture (ICLR 2025 · similarity 0.85)
- Fast attention mechanisms: a tale of parallelism (NeurIPS 2025 · similarity 0.85)
- Strassen Attention, Split VC Dimension and Compositionality in Transformers (NeurIPS 2025 · similarity 0.84)
- Ditto: Quantization-aware Secure Inference of Transformers upon MPC (ICML 2024 · similarity 0.84)
- Nimbus: Secure and Efficient Two-Party Inference for Transformers (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.