Research map / Architectures and efficiency / Pruning and efficient tuning

Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuning

ICML 2024 · Architectures and efficiency › Pruning and efficient tuning · approach: Adapter-based parameter-efficient tuning · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that randomly masking a pretrained LLM and fine-tuning only unmasked weights, with a tuned (often very large) learning rate, matches LoRA and full fine-tuning on SuperGLUE-style tasks with far fewer trainable parameters.

Problem

Parameter-efficient fine-tuning (PEFT) of large pretrained models still has many trainable parameters; the paper asks how far the parameter count can be reduced while keeping performance.

Approach

Random Masking: fix a random Bernoulli mask over pretrained weights and fine-tune only unmasked entries, using a much larger learning rate for sparser masks. Flatter loss landscapes (smaller Hessian norm) allow large learning rates, and sparse masks force more distant solutions that need them.

Key results

• On OPT-13b, Random Masking at 0.001% trainable parameters reaches 75.15 average vs 75.82 for full fine-tuning and 75.86 for LoRA (0.051%).
• Optimal learning rate for sparse masks (0.001% of parameters) reaches about 1e-1, a value that diverges for LoRA and other baselines.
• On OPT-1.3b, Random Masking at 0.1% reaches 70.88 average, comparable to LoRA at 70.81 with 0.120% parameters.

Measured effects reported in the paper

Surprising findings

• Random Masking with learning rates up to 1e-1 remains stable, although such learning rates typically diverge for standard PEFT methods.
• Structured (column-wise) masking performs worse and degrades faster than random masking as trainable parameters decrease.

Limitations

• Random Masking is not presented as state-of-the-art and may struggle on complex tasks requiring more expressive power (e.g., pretraining-like difficulty).
• Experiments use only 1000 training examples per task for most results and a small set of models; vision experiments use a different target module setup.
• Theory is limited to overparameterized linear regression with a masked-feature model.

Experimental setup

Datasets: SuperGLUE (BoolQ, CB, COPA, MultiRC, ReCoRD, RTE, WiC, WSC) SST-2 SQuAD DROP CIFAR10 GTSRB MNIST SVHN RESISC45

Benchmarks: SuperGLUE SST-2 SQuAD DROP CIFAR10 GTSRB MNIST SVHN RESISC45

Metrics: test accuracy F1

Baselines: full fine-tuning LoRA Adapter Prefix-Tuning BitFit AdaLoRA Structured Masking

Builds on: LoRA lottery ticket hypothesis BitFit intrinsic dimension

Closest papers by meaning

• Enhancing Large Language Model Performance with Gradient-Based Parameter Selection (AAAI 2025 · similarity 0.91)
• Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.89)
• PAT: Pruning-Aware Tuning for Large Language Models (AAAI 2025 · similarity 0.88)
• MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models (NeurIPS 2024 · similarity 0.88)
• Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark (ICML 2024 · similarity 0.87)
• LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning (NeurIPS 2024 · similarity 0.87)