Research map / Architectures and efficiency / Attention and transformers

Dense Structural Compression of Transformers via Gauge-Correct Channel Removal

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Vision transformer pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GaugeLasso, an additive symmetric group-lasso penalty with efficiency-scaled calibration and adaptive loss-target strength that physically removes transformer channels during training to produce dense, compressed architectures.

Problem

Reducing inference FMA and energy of dense transformers while keeping dense tensors for GPU throughput; product-norm penalties on gauge-connected factors are unstable under gauge freedom, and post-hoc pruning cannot reach the discovered structures.

Approach

Replace product-norm penalties on gauge-connected factor pairs with additive per-factor group-lasso penalties scaled by freed FMA per channel; at gauge balance the penalty equals a geometric-mean utility-per-FMA estimate, enabling calibrated trust-drop of channels during training with Sinkhorn gauge rebalancing and adaptive penalty strength tied to a loss target.

Key results

• On polynomial division over F31, converged seeds reach up to 255x compression (median 187x) with 1.000 exact match.
• On character-level LM at matched FMA, compressed model reaches 2.610 perplexity vs 2.652 for the uniform baseline.
• Post-hoc pruning with the same utility metric gives 3.191 perplexity on char LM (22% worse) and cannot compress FF31; from-scratch retraining fails on FF31 (0.161 EM vs 1.000).

Surprising findings

• Retraining a discovered architecture from scratch matches compressed quality on char LM and retrieval but fails on exact algorithmic FF31 execution.
• CoFi reimplementation stalls at 1.97x compression where 7.09x was requested.

Limitations

• Number of attention heads is fixed during training.
• Channel regrowth through nonlinearities is not supported.
• Experiments are small-scale (below 150M FMA) trained from scratch; not tested on pretrained LLMs.

Experimental setup

Datasets: PG-19 CelebA synthetic polynomial division over F31 synthetic modular-arithmetic retrieval

Benchmarks: PG-19 character-level perplexity CelebA masked autoencoding loss FF31 exact match retrieval accuracy

Metrics: compression ratio FMA exact match perplexity accuracy reconstruction loss step time

Baselines: CoFi post-hoc structured pruning uniform baseline transformer from-scratch retraining

Builds on: group lasso CoFi Muon Equi-normalization SwiGLU

Closest papers by meaning

• Linear Transformers Implicitly Discover Unified Numerical Algorithms (NeurIPS 2025 · similarity 0.87)
• CODA: Rewriting Transformer Blocks as GEMM-Epilogue Programs (NeurIPS 2026 · similarity 0.87)
• Adaptive Pruning of Pretrained Transformer via Differential Inclusions (ICLR 2025 · similarity 0.86)
• Learning Rate Transfer for Hybrid Transformer-SSM Architectures (NeurIPS 2026 · similarity 0.86)
• MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connections (ICML 2025 · similarity 0.86)
• The Quantization Benefits of Residual-Free Transformers (NeurIPS 2026 · similarity 0.86)