Research map / Architectures and efficiency / Pruning and efficient tuning
SS1: Accelerating Inference with Fast and Expressive Sketch Structured Transform
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Sketch Structured Transform (SS1), a randomized parameter-sharing linear layer that tiles weights within each neuron to cut FLOPs and speed GPU inference, and evaluates it on GPT-2, MLPMixer, BERT, Llama-3-8B and DLRM.
Problem
Standard linear layers dominate deep learning inference cost, but unstructured sparsity and randomized parameter sharing (RPS) reduce memory without reducing FLOPs or latency, while structured alternatives sacrifice expressivity or fail to speed up hardware execution.
Approach
SS1 restricts random parameter sharing to within a single neuron's weight vector, using K- and N-coalescing hash functions so that the input is first sketched (reduced by the compression factor via additions/subtractions) and then multiplied with the compressed weights, yielding a GPU-friendly reduction in FLOPs while keeping per-neuron independence for expressivity.
Key results
- SS1-2x GPT-2 Small reaches PPL 19.46 with 96M params vs. 20.43 for the original 124M model, and 1.3x throughput gains at larger sizes.
- Projected BERT-Base with SS1 and finetuning keeps GLUE 79.9 (66M params) vs. 82.16 for the baseline run, with 1.31x faster inference.
- Llama-3-8B with selective SS1 projection and no finetuning gives 1.11x wall-clock speedup, MMLU drops from 65.05 to 61.26.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace dense linear layers with SS1 (2x compression) when training GPT2-S | GPT2-S trained from scratch · wikitext-103 | PPL: 20.43 -> 19.458 | Helped |
| Increase SS1 compression from 2x to 8x (ablation) | GPT2-S trained from scratch · wikitext-103 | PPL: 19.458 -> 20.68 | Hurt |
| SS1 versus Monarch at comparable parameter counts | GPT2-S · wikitext-103 | PPL: SS1-2x 19.458 vs Monarch (nb=2) 19.606 at 96M vs 103M params | Helped |
| SS1 versus low-rank at comparable compression on CIFAR-100 MLPMixer | MLPMixer · CIFAR-100 | accuracy: SS1-8x 0.6068 vs Lowrank~8x 0.5353 (similar params) | Helped |
| Apply SS1 (2x) to MLPMixer vs dense on CIFAR-100 | MLPMixer · CIFAR-100 | accuracy: 0.6751 -> 0.6692 (2x); 0.6068 (8x) | Hurt |
| SS1 layers in GPT2-M/L end-to-end inference | GPT2 inference latency · wikitext-103 latency | model latency (ms): GPT2-L 376 -> 310.4 (4x); up to ~1.3x throughput | Helped |
| Projection of pretrained BERT to SS1 followed by finetuning | BERT-BASE · GLUE | GLUE score: 79.9 vs 82.16 baseline (our run) | No clear effect |
| Project selected Llama-3-8B layers to SS1 without finetuning | Llama-3-8B · MMLU | accuracy: 65.05 -> 61.26 | Hurt |
| Train DLRM MLPerf top MLP with SS1 (2x) from scratch | DLRM MLPerf · quality | quality: 0.8032 vs 0.8032 | No clear effect |
| Replace RPS (ROAST-MM) with SS1 at 2x compression on GPT2 (ablation) | GPT2-S · wikitext-103 | PPL: 19.87 -> 19.45; latency 238 -> 154 | Helped |
| Apply post-training quantization on top of SS1 models (ablation) | GPT-S SS1-4x and SS1-8x · wikitext-103 | PPL: 20.497 -> 20.53 (4x); 21.228 -> 21.26 (8x) | No clear effect |
| Learning rate tuning for Monarch baseline (ablation) | Monarch-8x · wikitext-103 | PPL: best PPL 20.65 vs 22.83 default; SS1-8x 20.13 at tuned LR | No clear effect |
Surprising findings
- Projected SS1 models perform reasonably well on Llama-3-8B even without any finetuning.
- Monarch inference latency is often worse than full dense multiplication at these shapes, contrary to its reported training gains.
Limitations
- Efficiency gains are marginal beyond 8x parameter compression.
- Theoretical analysis is limited to linear models and may not extend to deep networks.
- Backward pass cost is not reduced, so training latency benefits are not obtained; quantized SS1 kernels were not implemented.
Experimental setup
Datasets: WikiText-103 CIFAR-10 CIFAR-100 Tiny-ImageNet GLUE MMLU Winogrande DLRM MLPerf benchmark
Benchmarks: WikiText-103 CIFAR-10 CIFAR-100 GLUE MMLU Winogrande DLRM MLPerf
Metrics: perplexity accuracy latency throughput GLUE score
Baselines: Monarch LowRank ROAST-MM nn.Linear (original model) Small model
Builds on: Randomized parameter sharing (HashedNets / ROAST) Monarch low-rank factorization SmoothQuant
Closest papers by meaning
- An Efficient Matrix Multiplication Algorithm for Accelerating Inference in Binary and Ternary Neural Networks (ICML 2025 · similarity 0.86)
- Ladder-Residual: Parallelism-Aware Architecture for Accelerating Large Model Inference with Communication Overlapping (ICML 2025 · similarity 0.86)
- Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models (NeurIPS 2024 · similarity 0.85)
- DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMs (NeurIPS 2025 · similarity 0.85)
- Energy-Based Transformers are Scalable Learners and Thinkers (ICLR 2026 · similarity 0.85)
- LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.