Research map / Architectures and efficiency / Pruning and efficient tuning

APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference

ICML 2024 · Architectures and efficiency › Pruning and efficient tuning · approach: Adapter-based parameter-efficient tuning · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes APT, which jointly prunes and tunes pretrained LMs during fine-tuning using LoRA-style APT adapters with outlier-aware salience scoring and self-distillation, improving training and inference efficiency.

Problem

Fine-tuning and inference of large pretrained language models are expensive; PEFT reduces training memory but not inference cost, while structured pruning improves inference but increases training time and memory, and combining them hurts accuracy.

Approach

Adaptively prune LM blocks from the start of fine-tuning using an outlier-aware salience score computed from activations and gradients (avoiding frozen-weight gradients), while dynamically increasing LoRA-style tuning ranks in the most salient adapters, plus a lightweight self-distillation that shares frozen weights between teacher and student.

Key results

• On RoBERTa-base at 60% sparsity, APT reaches 86.4 MNLI and 94.5 SST2 with 592.1% training time (about 8x faster TTA than LoRA+Prune at 5128.3%) and 70.1% training memory.
• On LLaMA 2 7B at 30% sparsity, APT averages 50.0 across Open LLM tasks versus 45.5 for LoRA+Prune and 42.9 for LLMPruner, retaining about 86.4% of the dense model average.
• APT at 30% sparsity uses 75.8% of LoRA training memory and 67.2% of LoRA inference memory on LLaMA 2 7B.

Measured effects reported in the paper

Surprising findings

• Adaptive tuning without pruning (w/o A_P) on RoBERTa trains about 21% faster than full fine-tuning while using only 62.2% of its memory.
• On T5, APT prunes more decoder parameters, which yields worse inference speedup (1.3x) than the LoRA+Prune baseline (2.1x) despite higher task accuracy.

Limitations

• Pruned large LLaMA models show a larger performance gap, since distillation is skipped to save memory.
• Dynamic changes in parameter shapes can make training unstable; the optimizer must be reset after each size change.
• Low-rank adapters do not increase representation capacity; non-linear adapters are unexplored.

Experimental setup

Datasets: GLUE (SST2, MNLI, QQP, QNLI, CoLA, STS-B, MRPC, RTE) SQuAD v2.0 CNN/DM Alpaca (GPT-4 generated)

Benchmarks: GLUE (MNLI, SST2) SQuAD v2.0 CNN/DM ARC HellaSwag MMLU TruthfulQA

Metrics: accuracy F1 ROUGE training time to accuracy training peak memory inference time inference memory

Baselines: LoRA LoRA+Prune Prune+Distill (CoFi) LoRA+Prune+Distill LLMPruner PST LRP MaP MvP

Builds on: LoRA CoFi AdaLoRA Mask Tuning LLM-Pruner

Closest papers by meaning

• PAT: Pruning-Aware Tuning for Large Language Models (AAAI 2025 · similarity 0.93)
• SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models (ICML 2024 · similarity 0.92)
• Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024 · similarity 0.92)
• Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.91)
• The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.91)
• La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation (ICML 2025 · similarity 0.90)