Research map / Architectures and efficiency / Mixture of experts

FourierMoE: Fourier Mixture-of-Experts Adaptation of Large Language Models

NeurIPS 2026 · Architectures and efficiency › Mixture of experts · approach: MoE LLM compression · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FourierMoE, a mixture-of-experts PEFT method where each expert learns conjugate-symmetric complex Fourier coefficients mapped to real weight updates via IDFT, with a frequency-adaptive router, evaluated on commonsense, math, vision and GLUE tasks.

Problem

Standard PEFT methods and spatial-domain mixture-of-PEFT-experts (MoPE) suffer from task interference, limited parameter budgets, and structural redundancy in multi-task LLM fine-tuning.

Approach

Reparameterize weight updates in the spectral domain: a frequency-adaptive router dispatches tokens to experts that each hold complex Fourier coefficients restricted to a Gaussian bandpass frequency region, with conjugate symmetry enforced so the IDFT yields real-valued weight updates losslessly.

Key results

• On RoBERTa-large GLUE single-task, FourierMoE averages 91.40 with 0.42% trainable parameters, versus 89.76 for the best MoPE baseline GOAT and 86.73 for FourierFT.
• On LLaMA-2 7B commonsense reasoning (multi-task), FourierMoE averages 83.95 vs. 82.73 for GOAT while using 0.06% parameters (reported as 16x fewer than GOAT).
• On LLaMA-3 8B math reasoning, FourierMoE averages 73.24, a 3.01% gain over DoRA and 3.61% over HydraLoRA.

Surprising findings

• Ablation of the imaginary part and the conjugate symmetry constraint both degrade performance, while the paper reports that wider frequency bandwidth beyond 0.12 hurts accuracy.

Limitations

• Inference latency and training time are higher than plain LoRA (313 ms vs 53.4 ms; 58 s vs 13.13 s per epoch).
• Hyperparameters such as frequency bandwidth and number of coefficients require per-task tuning; performance is sensitive to bandwidth.

Experimental setup

Datasets: Commonsense170K Math10K GSM8K AQuA CoLA SST-2 MRPC QQP MNLI QNLI RTE Cars

Benchmarks: BoolQ PIQA SIQA HellaSwag WinoGrande ARC-e ARC-c OBQA GSM8K SVAMP MultiArith AddSub

Metrics: accuracy trainable parameter fraction training FLOPs training and inference latency GPU memory

Baselines: FFT FFT MoE LoRA rsLoRA DoRA PiSSA MiLoRA KaSA LoRA-Dash NEAT TopLoRA MELoRA

Builds on: FourierFT LoRA HydraLoRA GOAT MoLoRA

Closest papers by meaning

• From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs (NeurIPS 2026 · similarity 0.91)
• RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanism (NeurIPS 2026 · similarity 0.89)
• Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models (ICLR 2024 · similarity 0.89)
• Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.89)
• LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing (NeurIPS 2026 · similarity 0.88)
• MoLEx: Mixture of Layer Experts for Fine-tuning with Sparse Upcycling (ICLR 2025 · similarity 0.88)