Research map / Architectures and efficiency / Pruning and efficient tuning

Joint Fine-tuning and Conversion of Pretrained Speech and Language Models towards Linear Complexity

ICLR 2025 · Architectures and efficiency › Pruning and efficient tuning · approach: Low-rank compression and merging · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CALD, which jointly converts a pretrained transformer (RoBERTa, Pythia, Wav2Vec2) into a linear-complexity model (Linformer, Mamba, Mamba2) and fine-tunes it on the target task via layerwise distillation with several guidance modes.

Problem

Linear-complexity transformer substitutes (Linformer, Mamba, Mamba2) usually lack pretrained checkpoints, especially for speech, and re-pretraining them is costly; converting existing pretrained transformers during fine-tuning is a cheaper alternative.

Approach

Cross-Architecture Layerwise Distillation: initialize the student with transferred non-attention parameters and distill hidden states after the feed-forward layer from a target teacher, with extra guidance modes (trajectory-guided teacher updates, waypoint checkpoints, and a hybrid of guided then unguided training) to retain pretrained knowledge.

Key results

• On GLUE-style tasks converting RoBERTa to Linformer, trajectory-guided CALD reaches 92.5% average accuracy (+0.2% over the re-pretrained Linformer at 92.3%), while unguided conversion drops to 80.0%.
• For Wav2Vec2 to Mamba2 speech conversion, target-guided CALD gets 6.56% WER on TED-LIUM vs 6.24% for standard Wav2Vec2 and 11.29% unguided; hybrid reaches 6.41% WER and 91.23% SLURP IC accuracy.
• On Pythia-1B to Mamba conversion with 2% Pile (6.0B tokens), hybrid CALD reaches 0.525 average zero-shot accuracy vs 0.514 unguided and 0.544 for the original Pythia-1B.

Measured effects reported in the paper

Surprising findings

• Waypoint-guided distillation was not helpful or even harmful for speech tasks, because Wav2Vec2 hidden states shift strongly early in fine-tuning (cosine distance 0.372 after 2,000 steps), unlike RoBERTa.
• Output (KD) distillation led to suboptimal results in language processing and language modeling, so the KD weight was set to zero there.

Limitations

• Language modeling experiments are limited to Pythia-1B on a tiny subset of the Pile due to compute, and show a remaining gap to the original model.
• Unguided training is unstable and fails to converge for large Mamba2 speech models trained from scratch or with transferred parameters.
• Inference speed comparison is limited; the actual speedup depends on implementation and was not investigated in depth.

Experimental setup

Datasets: QNLI QQP SST2 IMDB Pile TED-LIUMv3 SLURP VoxCeleb1

Benchmarks: QNLI QQP SST2 IMDB Lambada PIQA Winogrande WSC ARC-Easy ARC-Challenge SciQ LogiQA

Metrics: accuracy word error rate zero-shot accuracy

Baselines: Unguided parameter transfer Re-pretrained Linformer Standard fine-tuned RoBERTa Standard fine-tuned Wav2Vec2 Pythia-1B

Builds on: Linformer Mamba Mamba2 knowledge distillation Wav2Vec2

Closest papers by meaning

• Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.88)
• Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning (ICLR 2024 · similarity 0.88)
• MISA: Memory-Efficient LLMs Optimization with Module-wise Importance Sampling (NeurIPS 2025 · similarity 0.88)
• Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning (NeurIPS 2025 · similarity 0.88)
• Enhancing Large Language Model Performance with Gradient-Based Parameter Selection (AAAI 2025 · similarity 0.88)
• Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA (ICLR 2025 · similarity 0.88)