Research map / Continual learning / Continual fine-tuning of LLMs

TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by a Hierarchical Gradient-Similarity Tree

ICML 2025 · Continual learning › Continual fine-tuning of LLMs · approach: Continual pretraining of language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TreeLoRA, which organizes per-task layer-wise LoRA adapters into a hierarchical tree built from gradient similarity, using LCB bandit search and sparse updates for efficient continual fine-tuning of ViTs and LLMs.

Problem

Continual fine-tuning of large pre-trained models is computationally costly because existing methods require gradients or computation over all previous tasks, scaling linearly with task count; task identity is unknown at test time.

Approach

Builds a hierarchical tree of task groups by gradient similarity (L1 distance), searches it on the fly with a lower-confidence-bound multi-armed bandit that needs only one previous-gradient query per round, and updates only sparse, layer-wise LoRA parameters, claiming regret reduced from O(sqrt(N)) to O(log N) under a smooth-tree assumption.

Key results

• On Split CIFAR-100 with ViT, TreeLoRA reaches 88.54% accuracy vs 88.46% for HiDeLoRA, with training time 214 s vs 692 s for HiDeLoRA.
• On TRACE with Mistral-7B-Instruct-v0.3, TreeLoRA achieves OP 54.77% and BWT 3.77%, versus 52.02% OP for O-LoRA.
• Reported speedups of up to 3.2x for ViTs and 2.4x for LLMs in training time relative to prior state-of-the-art methods.

Measured effects reported in the paper

Surprising findings

• Reduced training epochs for TreeLoRA (e.g., 2 epochs) still reach 87.73% on Split CIFAR-100, while other methods degrade under the same reduced epoch budget (e.g., O-LoRA drops from 33.73% to 30.60% on Gemma-2B-it).

Limitations

• Theoretical regret bound relies on a smooth-tree assumption and an unbiased-noise assumption that are not verified empirically.
• Tree depth and storage-budget merging are hand-set hyperparameters; depth is capped by the number of transformer layers.

Experimental setup

Datasets: Split CIFAR-100 Split ImageNet-R Split CUB-200 TRACE ScienceQA NumGLUE

Benchmarks: Split CIFAR-100 Split ImageNet-R Split CUB-200 TRACE Math-LLM O-LoRA task orders

Metrics: accuracy overall performance (OP) backward transfer (BWT) training time (s)

Baselines: SeqLoRA GEM EWC L2P DualPrompt HiDePrompt HiDeLoRA O-LoRA OGD FIX (ICL) SAPT TASL

Builds on: LoRA O-LoRA HiDeLoRA HiDePrompt lower confidence bound bandit

Closest papers by meaning

• C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models (NeurIPS 2026 · similarity 0.89)
• PaLoRA: Paced Low-Rank Adaptation for Continual Learning (NeurIPS 2026 · similarity 0.88)
• SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning (ICLR 2025 · similarity 0.88)
• CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning (CVPR 2025 · similarity 0.88)
• Learn more, but bother less: parameter efficient continual learning (NeurIPS 2024 · similarity 0.87)
• TinySubNets: An Efficient and Low Capacity Continual Learning Strategy (AAAI 2025 · similarity 0.87)