Research map / Architectures and efficiency / Pruning and efficient tuning

MARS: Harmonizing Multimodal Convergence via Adaptive Rank Search

NeurIPS 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: LoRA variants and theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MARS, which uses two fitted scaling laws (performance and per-module convergence time) to prune and select modality-specific LoRA rank pairs for VLM fine-tuning, reporting gains over LR tuning and adaptive-rank baselines at ~11.5x lower search cost.

Problem

LoRA fine-tuning of multimodal LLMs suffers from imbalanced convergence between vision encoder and LLM, causing suboptimal accuracy; heuristic learning-rate tuning is costly and naive rank-pair search requires full fine-tuning per candidate.

Approach

Use LoRA rank as a per-modality controller of convergence speed: dual scaling laws (Scaling Law-C for module convergence steps, Scaling Law-P for perplexity) prune rank pairs to those with aligned convergence, then pick the pair with lowest predicted perplexity.

Key results

• On LLaVA-OV-7B, MARS reaches 74.25% ScienceQA accuracy vs 72.26% for the best differential-learning-rate LoRA setting.
• MARS achieves lower LLaVA Bench perplexity than AdaLoRA and GeoLoRA across all four models (e.g., 2.1875 vs 2.4888 for GeoLoRA on LLaVA-OV-7B).
• MARS reduces total search and fine-tuning time by more than 11.5x on average versus naive exhaustive rank search.

Surprising findings

• Simply assigning higher LoRA rank to the vision encoder does not invariably improve performance; the optimal VE-LLM rank pair depends on dataset size and the other module's rank.
• Adaptive rank methods designed for unimodal models (AdaLoRA, GeoLoRA) underperform even full-rank tuning on the multimodal benchmarks.

Limitations

• Scaling laws are calibrated on from-scratch LLaVA-OV variants with a single data source (LLaVA-158K), limiting generality claims.
• Rare edge cases (0.72% of runs) yield complex-valued ranks, requiring a fallback to equal ranks.
• Evaluation on the main tables uses a small ScienceQA test subset (500 samples), and results are from single runs.

Experimental setup

Datasets: LLaVA-158K LLaVA-1.5-665K ScienceQA LLaVA Bench

Benchmarks: LLaVA Bench ScienceQA MME POPE MMStar TextCaps AI2D GQA

Metrics: validation perplexity accuracy F1 ROUGE-L

Baselines: AdaLoRA GeoLoRA Full-rank tuning fixed-rank LoRA with differential learning rates fixed differential LoRA rank pairs

Builds on: LoRA AdaLoRA GeoLoRA LLaVA-OneVision neural scaling laws

Closest papers by meaning

• SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.89)
• Dynamics-Aware Preference Optimization for Vision-Language Models (CVPR 2026 · similarity 0.89)
• CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models (CVPR 2026 · similarity 0.89)
• Frustratingly Easy Test-Time Adaptation of Vision-Language Models (NeurIPS 2024 · similarity 0.88)
• Connecting the Dots: Collaborative Fine-tuning for Black-Box Vision-Language Models (ICML 2024 · similarity 0.88)
• Mordal: Automated Pretrained Model Selection for Vision Language Models (ICLR 2026 · similarity 0.88)