Research map / Architectures and efficiency / Pruning and efficient tuning

IterIS: Iterative Inference-Solving Alignment for LoRA Merging

CVPR 2025 · Architectures and efficiency › Pruning and efficient tuning · approach: LoRA variants and theory · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes IterIS, an iterative inference-solving, closed-form LoRA merging method that refines the merging objective using actual unified-adapter input features, with regularization and adaptive weights, evaluated on diffusion, VLM, and LLM tasks.

Problem

Merging multiple task-specific LoRAs into one unified adapter without training data or gradient-based training; existing real-distribution merging methods approximate unified-adapter inputs with each LoRA's inputs, need many samples, and produce unbalanced objectives.

Approach

Replace per-LoRA input features with the features actually seen by the unified adapter, recomputed iteratively via inference and closed-form solves; add a Frobenius-scaled identity regularizer so only a few dozen samples are needed, and adaptive per-task weights (inverse squared norms) to balance objective terms.

Key results

• On multi-style captioning (POS+NEG), IterIS reaches 83.1% POS and 78.1% NEG style accuracy versus 62.4%/69.2% for RegMean.
• On seven diffusion concept pairs, IterIS beats linear merging on average by 0.78% (image alignment 1), 1.61% (image alignment 2), and 2.53% (text alignment).
• On GLUE multi-task integration (21 pairs, FLAN-T5-base), IterIS averages 5.811 vs 5.764 for RegMean and 5.625 for linear merging.

Measured effects reported in the paper

Surprising findings

• Linear merging of the POS and NEG caption LoRAs is close to chance on style accuracy (52.2% and 55.7%), and task arithmetic yields the same accuracy as linear merging.
• On COLA, IterIS slightly underperforms linear merging (0.666 vs 0.678).

Limitations

• Iteration count is capped at 20 to prevent overfitting, with convergence bound only derived for self-attention encoders with Q, K, V LoRAs.
• Evaluation is on small models (Stable Diffusion v1.5, BLIP base, FLAN-T5) and a limited set of task pairs.
• Only the authors' own two-to-four-task combinations are evaluated; the method is not shown to generalize to other PEFT methods (acknowledged as future work).

Experimental setup

Datasets: CustomConcept101 DreamBooth SentiCap Emotion datasets (Emoint, EC, TEC, ISEAR) GLUE

Benchmarks: CustomConcept101 DreamBooth SentiCap Emotion datasets GLUE

Metrics: CLIP image alignment CLIP text alignment style accuracy CIDEr BLEU-1 to BLEU-4 F1 accuracy MCC

Baselines: Linear merging Textual inversion Custom Diffusion Task arithmetic Ties merging Hyper-linear RegMean

Builds on: RegMean Mix-of-Show Custom Diffusion Linear merging (PEMs) DreamBooth-LoRA

Closest papers by meaning

• Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering (ICLR 2025 · similarity 0.90)
• LoRA-GA: Low-Rank Adaptation with Gradient Approximation (NeurIPS 2024 · similarity 0.90)
• Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment (ICML 2025 · similarity 0.90)
• Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.89)
• SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.89)
• LoRAtorio: An intrinsic approach to LoRA Skill Composition (NeurIPS 2026 · similarity 0.89)