Research map / Architectures and efficiency / Pruning and efficient tuning

Mixture-of-Control: State-Aware Fine-Tuning for Transformer-based Models

NeurIPS 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: Adapter-based parameter-efficient tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Mixture-of-Control (MoC), a state-based fine-tuning method that routes block-wise low-rank control modules as sparse MoE experts to enable cross-block communication without extra pretrained-block passes.

Problem

State-based fine-tuning methods apply only block-local control updates, limiting inter-block information exchange, while existing cross-block mechanisms (e.g., MoLEx) add substantial compute and memory overhead.

Approach

Treat each block's low-rank control module as an expert in a sparse MoE with a shared gate, and mix the top-K routed global control signal with the local control (weight alpha vs 1-alpha) injected into the frozen block, enabling cross-depth communication at low overhead.

Key results

• On commonsense reasoning with LLaMA3-8B, MoC averages 86.7% vs 85.3% for Parallel Control and 85.2% for DoRA, with 56.7GB GPU memory vs 54.0GB for Control.
• On GLUE with RoBERTa-Large, MoC averages 89.8 vs 88.6 for Control and 87.8 for LoRA.
• On Qwen2.5-14B commonsense reasoning, MoC reaches 90.1% average vs 87.7% for LoRA, while MoLEx uses 124.8GB GPU memory vs 61.3GB for MoC.

Surprising findings

• Top-K=1 was reported to perform best in one ablation text, while the TopK figure reportedly shows two activated experts best, so the configuration is internally inconsistent.
• Gains over the strongest baseline are modest (1.0-3.0% on NLG), and MoC's inference throughput remains below merged Weight-FT methods.

Limitations

• Theoretical analysis relies on strong assumptions (Lipschitz bounds, pretrained mixing closeness) and some proof steps are heuristic.
• State-based methods remain less efficient in inference than mergeable weight-based fine-tuning.
• Experiments limited to a few backbones and benchmarks; no variance reporting beyond medians for GLUE.

Experimental setup

Datasets: BoolQ PIQA SIQA HellaSwag WinoGrande ARC-e ARC-c OBQA GLUE

Benchmarks: BoolQ PIQA SIQA HellaSwag WinoGrande ARC-e ARC-c OBQA GLUE

Metrics: accuracy Matthews correlation Pearson correlation training time (h) GPU memory (GB) inference throughput

Baselines: LoRA DoRA MoLEx Parallel Control (Control)

Builds on: Parallel Control (State-Based Framework) MoLEx LoRA Sparse Mixture of Experts

Closest papers by meaning

• WeGeFT: Weight‑Generative Fine‑Tuning for Multi‑Faceted Efficient Adaptation of Large Models (ICML 2025 · similarity 0.87)
• Computational Limits of Low-Rank Adaptation (LoRA) Fine-Tuning for Transformer Models (ICLR 2025 · similarity 0.86)
• MambaPEFT: Exploring Parameter-Efficient Fine-Tuning for Mamba (ICLR 2025 · similarity 0.86)
• Optimal Control for Transformer Architectures: Enhancing Generalization, Robustness and Efficiency (NeurIPS 2025 · similarity 0.86)
• Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models (ICLR 2025 · similarity 0.85)
• From Shortcut to Induction Head: How Data Diversity Shapes Algorithm Selection in Transformers (NeurIPS 2025 · similarity 0.85)