Research map / Architectures and efficiency / Pruning and efficient tuning

QuadEnhancer: Leveraging Quadratic Transformations to Enhance Deep Neural Networks

NeurIPS 2025 · Architectures and efficiency › Pruning and efficient tuning · approach: Low-rank compression and merging · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes QuadEnhancer, a lightweight quadratic enhancer that adds band-sparse, weight-shared quadratic interactions to every linear layer, and reports gains on ViT image classification, GPT-2 text classification, and LoRA fine-tuning of LLaMA models.

Problem

Standard quadratic (polynomial) transformations improve expressiveness but add O(n^2) or O(dn^2) parameters and computation, so they are rarely used in practical deep networks.

Approach

Replace each linear layer's output y = Wx with a quadratic form z = (Λy)⊙y + y + b, reusing the linear response y = Wx (weight sharing), and restricting Λ to a band matrix with shift set K={1} (rolled copies of y), so only about k*d extra parameters and 2(k+1)d extra FLOPs are added.

Key results

• ViT-M+QE improves average accuracy over the six downstream datasets plus ImageNet from 82.44% to 83.91% with about 0.02M extra parameters.
• LLaMA2-7B with LoRA/16+QE averages 80.25% on commonsense reasoning versus 77.61% for LoRA/32 with 27.6M vs 53.5M trainable parameters.
• GPT-2/16+QE reduces WikiText-2 perplexity from 4.90 to 4.81 with 0.82M vs 0.71M parameters.

Measured effects reported in the paper

Surprising findings

• Using LoRA/16+QE with half the trainable parameters outperforms LoRA/32 without QE on LLaMA-7B and LLaMA2-7B commonsense reasoning.
• Excluding the self-interaction shift (r=0) is motivated by heavier tails of square terms, which the authors argue cause FP16 instability.

Limitations

• No error bars or repeated runs; the authors state they lacked time for repeated experiments.
• Experiments are small-scale proof-of-concept (ViT-sized models, WikiText-2 pre-training), with limited scaling evidence.
• Quadratic enhancer adds some computational overhead and slower early convergence compared to baselines.

Experimental setup

Datasets: ImageNet-1k Caltech CIFAR-10 CIFAR-100 Flowers Food Pets WikiText-2 IMDB Yelp AG-News SST-2

Benchmarks: ImageNet-1k Caltech CIFAR-10 CIFAR-100 Flowers Food Pets WikiText-2 IMDB Yelp AG-News SST-2

Metrics: accuracy perplexity parameter count FLOPs

Baselines: ViT-M ViT-XT ViT-T GPT-2/16 GPT-2/32 LoRA QuadraNet SwiGLU

Builds on: LoRA QuadraNet SwiGLU

Closest papers by meaning

• Q3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Training (NeurIPS 2025 · similarity 0.85)
• AffineQuant: Affine Transformation Quantization for Large Language Models (ICLR 2024 · similarity 0.85)
• Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models (NeurIPS 2024 · similarity 0.84)
• An Efficient Matrix Multiplication Algorithm for Accelerating Inference in Binary and Ternary Neural Networks (ICML 2025 · similarity 0.84)
• GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling (NeurIPS 2025 · similarity 0.84)
• SpinQuant: LLM Quantization with Learned Rotations (ICLR 2025 · similarity 0.84)