Research map / Architectures and efficiency / Pruning and efficient tuning
DRONE: Data-aware Low-rank Compression for Large NLP Models
Patrick CHen, Hsiang-Fu Yu, Inderjit S Dhillon, Cho-Jui Hsieh
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace SVD with data-aware low-rank decomposition at same ranks (ablation) | BERT-base · GLUE MRPC | F1: 63.8 -> 86.7 | Improved |
| replace SVD with data-aware low-rank decomposition at same ranks (ablation) | BERT-base · GLUE CoLA | Matthew's correlation: 12.4 -> 52.5 | Improved |
| add 1-epoch fine-tuning after DRONE (ablation) | DRONE · GLUE MRPC | F1: 86.7 -> 88.0 | Improved |
| add 1-epoch fine-tuning after DRONE (ablation) | DRONE · GLUE MNLI | accuracy: 82.0 -> 82.6 | Improved |
| use DRONE instead of fine-tuned SVD (ablation) | SVD-Retrain · GLUE CoLA | Matthew's correlation: 24.4 -> 53.2 | Improved |
| combine DRONE with TinyBERT distillation | TinyBERT · GLUE SST-2 | CPU speedup: 11.7x -> 15.3x | Improved |
| combine DRONE with TinyBERT distillation | TinyBERT · GLUE STS-B | accuracy: 86.9 -> 87.0 | No clear effect |
| DRONE (no attention) vs structured pruning | BERT-base · GLUE MRPC | accuracy: 88.61 -> 89.4 | Improved |
| use DRONE instead of SVD as low-rank init before training | RTE training · GLUE RTE | accuracy: 52.1 -> 72.9 | Improved |
| DRONE low-rank at same rank as inference setting before training | RTE training · GLUE RTE | accuracy: 72.6 -> 68.2 | Hurt |
| apply DRONE to machine translation transformer | 2-layer Transformer · en-de | BLEU: 33.47 -> 33.26 | No clear effect |
Closest papers by meaning
- SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression (AAAI 2025 · similarity 0.85)
- Compressing Large Language Models using Low Rank and Low Precision Decomposition (NeurIPS 2024 · similarity 0.85)
- MoDeGPT: Modular Decomposition for Large Language Model Compression (ICLR 2025 · similarity 0.84)
- Language Modeling Is Compression (ICLR 2024 · similarity 0.83)
- Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Models (NeurIPS 2024 · similarity 0.83)
- Efficient Hybrid Language Model Compression through Group-Aware SSM Pruning (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
