Research map / Architectures and efficiency / Quantization

Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edge

AAAI 2024 · Architectures and efficiency › Quantization · approach: Post-training quantization of LLMs · 35 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Agile-Quant quantizes both weights (4-bit via GPTQ) and activations (8/4-bit) of LLaMA, OPT and BLOOM, adds activation-aware token pruning and a TRIP per-channel refinement, and implements INT4/INT8 kernels on edge CPUs for up to 2.55x on-device speedup over FP16.

Problem

Activation quantization of LLMs is hard because of outliers, and commodity edge CPUs/SIMD units do not efficiently execute sub-8-bit activation x weight matrix multiplications, so inference acceleration on edge devices is limited.

Approach

Combine GPTQ-style 4-bit weight quantization with 8/4-bit activation quantization guided by attention-based token pruning (using attentivity to the start token) and a TRIP channel-wise power-of-two refinement factor, plus a SIMD-based 4-bit multiplier implemented in ARM kernels for edge devices.

Key results

• Agile-Quant-8 with token pruning reaches LLaMA-7B WikiText-2 perplexity 6.09 vs FP16 5.68 and GPTQ INT4 weight-only 5.85.
• On edge CPUs, token-pruned 8-bit activation models achieve about 1.7x to 1.8x speedup over FP16 (e.g., LLaMA-7B on Snapdragon 870 from 10.6 s to 5.89 s).
• Agile-Quant-4 reaches up to about 2.4x to 2.5x speedup over FP16 on Snapdragon 870 and Raspberry Pi 4B, at a perplexity cost (LLaMA-7B 8.81 WikiText-2).

Measured effects reported in the paper

Surprising findings

• Token pruning, applied to already quantized activations, lowers perplexity relative to the dense quantized model for LLaMA, OPT and BLOOM in several settings.
• Token pruning helps only when token sparsity is small and only for sufficiently long sequences (e.g., 2048 tokens); it hurts at high sparsity or short sequences.

Limitations

• 4-bit activation variant incurs notable perplexity degradation (e.g., LLaMA-7B 8.81 vs 5.68 FP16).
• Hardware evaluation limited to an Arm CPU (Snapdragon 870) and Raspberry Pi 4B; kernels built on ArmComputeLibrary v22.05.
• Token pruning benefit only holds for long input sequences and small sparsity.

Experimental setup

Datasets: WikiText-2 C4 PTB

Benchmarks: WikiText-2 C4 PTB

Metrics: perplexity latency (s) model size (GB)

Baselines: RTN GPTQ AWQ SpQR SqueezeLLM MoFQ8 ZeroQuant-FP ZeroQuant-V2

Builds on: GPTQ ZeroQuant-V2 GEMMLOWP ArmComputeLibrary

Closest papers by meaning

• ConQuR: Corner Aligned Activation Quantization via Optimized Rotations for LLMs (NeurIPS 2026 · similarity 0.90)
• BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inference (ICML 2025 · similarity 0.89)
• ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models (AAAI 2025 · similarity 0.89)
• DP-LLM: Runtime Model Adaptation with Dynamic Layer-wise Precision Assignment (NeurIPS 2025 · similarity 0.89)
• ScaleBITS: Scalable Bitwidth Search for Hardware-Aligned Mixed-Precision LLMs (NeurIPS 2026 · similarity 0.89)
• ASER: Activation Smoothing and Error Reconstruction for Large Language Model Quantization (AAAI 2025 · similarity 0.89)