Research map / Architectures and efficiency / Attention and transformers
DiJiang: Efficient Large Language Models through Compact Kernelization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Converts pre-trained softmax-attention Transformers into linear-complexity models via frequency-domain (DCT-based) weighted quasi-Monte Carlo kernelization, requiring only light fine-tuning.
Problem
Quadratic attention cost in Transformers, and the fact that existing linear-attention methods require costly retraining, which is impractical for large language models.
Approach
Approximates the Gaussian/softmax attention kernel with positive fixed features built from quasi-Monte Carlo points, adds a learnable per-feature weight, and computes the random-feature projection with a DCT so that a pre-trained vanilla Transformer can be converted to linear attention with little fine-tuning.
Key results
- DiJiang-7B averages 0.557 vs LLaMA2-7B 0.565 across 11 benchmarks while using 40B tokens instead of 2000B (about 1/50 training cost).
- DiJiang-410M averages 0.456 vs Pythia-410M 0.454 on six benchmarks, with training time of 6.6 days vs 105.8 days.
- In the 410M fine-tuning comparison, DiJiang (0.4567) outperforms Performer (0.4183), RetNet (0.3843), Linformer and Cosformer.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Pythia-70M vanilla attention with DiJiang attention | Pythia-70M · Pythia-70M evaluation average | Avg: 0.375 -> 0.389 | Helped |
| Replace Pythia-160M vanilla attention with DiJiang attention | Pythia-160M · Pythia-160M evaluation average | Avg: 0.391 -> 0.398 | Helped |
| Replace Pythia-410M vanilla attention with DiJiang attention | Pythia-410M · Pythia-410M evaluation average | Avg: 0.454 -> 0.456 | Helped |
| Replace Pythia-1B vanilla attention with DiJiang attention | Pythia-1B · Pythia-1B evaluation average | Avg: 0.456 -> 0.440 | Hurt |
| Replace Pythia-2.8B vanilla attention with DiJiang attention | Pythia-2.8B · Pythia-2.8B evaluation average | Avg: 0.478 -> 0.473 | Hurt |
| Replace vanilla attention with DiJiang attention | Pythia-70M · Pythia-70M fine-tuning | Training time (day): 21.3 -> 1.3 | Helped |
| Replace vanilla attention with DiJiang attention | Pythia-70M · Pythia-70M, token length 2048 | Inference throughput (tokens/s): 2037 -> 2605 | Helped |
| Use weighted frequency-domain features rather than PFF (ablation) | DiJiang on Pythia-410M · Pythia-410M fine-tuning, six benchmarks | Avg: 0.4264 -> 0.4567 | Helped |
| Replace Pythia-410M vanilla attention with DiJiang | Pythia-410M · Pythia-410M fine-tuning, six benchmarks | Avg: 0.454 -> 0.4567 | Helped |
| Replace LLaMA2-7B with fine-tuned DiJiang-7B | LLaMA2-7B · LLaMA2-7B benchmark average | Avg: 0.565 -> 0.557 | Hurt |
| Fine-tune using DiJiang rather than original-scale training | LLaMA2-7B · LLaMA2-7B training | Training tokens: 2000B -> 40B | Helped |
Surprising findings
- Converting to linear attention with DCT-based features and light fine-tuning nearly matches the original Transformer, while retraining-based linear models like RetNet score lower despite low training loss.
- Some DiJiang-350M and TinyLLaMA-1.1B per-task results are inconsistent with the average, e.g., DiJiang-7B scores 0.346 on SIQA versus 0.485 for LLaMA2-7B.
Limitations
- Benchmark results for DiJiang-7B and some small models are mixed, with large per-task drops (e.g., SIQA) and the paper uses m=d to avoid extra cost.
- Evaluation is limited to a few zero-shot commonsense benchmarks and does not cover long-context or generation quality beyond qualitative examples.
- Hardware and total compute for training are only partially stated (A800 timing), and the training data for 7B is the Pile rather than the original corpus.
Experimental setup
Datasets: The Pile
Benchmarks: PIQA WinoGrande WSC ARC-E ARC-C LogiQA SIQA BoolQ HellaSwag MMLU NQ COPA
Metrics: accuracy average accuracy training time (day) inference throughput (tokens/s) perplexity/loss curves
Baselines: Pythia Linformer Cosformer Performer RetNet OPT-350M TinyLLaMA-1.1B LLaMA2-7B Mamba-1.3B
Builds on: Performer Linear Transformer RetNet Bochner's theorem
Closest papers by meaning
- Efficient Large Language Model Inference with Neural Block Linearization (NeurIPS 2025 · similarity 0.86)
- PolySketchFormer: Fast Transformers via Sketching Polynomial Kernels (ICML 2024 · similarity 0.86)
- Joint Fine-tuning and Conversion of Pretrained Speech and Language Models towards Linear Complexity (ICLR 2025 · similarity 0.86)
- Building on Efficient Foundations: Effective Training of LLMs with Structured Feedforward Layers (NeurIPS 2024 · similarity 0.86)
- MemoryFormer : Minimize Transformer Computation by Removing Fully-Connected Layers (NeurIPS 2024 · similarity 0.85)
- Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.