Research map / Time series / Foundation models for time series
Filtered not Mixed: Filtering-Based Online Gating for Mixture of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoE-F, an online gating mechanism that combines N pre-trained LLM or time-series experts via parallel Wonham-Shiryaev-style stochastic filters and a Gibbs-softmin robust aggregation, evaluated on NIFTY financial movement classification and ETT long-horizon forecasting.
Problem
Static gating in mixture-of-experts ignores temporal structure; for online time-series prediction, there is no principled way to adaptively weight multiple pre-trained experts (LLMs or forecasters) as their running performance evolves.
Approach
Model the best-expert indicator as a finite-state continuous-time hidden Markov chain observed through each expert's running loss; run N parallel closed-form Wonham-Shiryaev filters, then aggregate their predictions with a Gibbs/softmin weighting and re-estimate the intensity matrix Q from the aggregated weights.
Key results
- On NIFTY test split, MoE-F reaches F1 0.52 vs best single expert ~0.35 (17 points absolute, 48.5% relative).
- With LoRA-adapted Llama-2-7b experts, MoE-F reaches F1 0.43 vs 0.22 base.
- On ETTh1/ETTh2 long-horizon forecasting, MoE-F using only three forecasters (DLinear, PatchTST, SparseTSF) attains competitive MSE, e.g. 0.3630 at horizon 96 on ETTh1, matching best listed SOTA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply MoE-F online filtering over the seven LLM experts | LLM experts on NIFTY · NIFTY test split | F1: 0.35 -> 0.52 | Helped |
| Apply MoE-F filtering over DLinear, PatchTST, and SparseTSF | Time-series forecasting experts · ETTh1, horizon 96 | MSE: 0.3631 -> 0.3630 | Helped |
| Apply MoE-F filtering over DLinear, PatchTST, and SparseTSF | Time-series forecasting experts · ETTh1, horizon 192 | MSE: 0.4000 -> 0.4165 | Hurt |
| Apply MoE-F filtering over DLinear, PatchTST, and SparseTSF | Time-series forecasting experts · ETTh1, horizon 336 | MSE: 0.4346 -> 0.4178 | Helped |
| Apply MoE-F filtering over DLinear, PatchTST, and SparseTSF | Time-series forecasting experts · ETTh1, horizon 720 | MSE: 0.4238 -> 0.4157 | Helped |
| Apply MoE-F filtering over DLinear, PatchTST, and SparseTSF | Time-series forecasting experts · ETTh2, horizon 96 | MSE: 0.274 -> 0.2911 | Hurt |
| Apply MoE-F filtering over DLinear, PatchTST, and SparseTSF | Time-series forecasting experts · ETTh2, horizon 192 | MSE: 0.333 -> 0.3294 | Helped |
| Apply MoE-F filtering over DLinear, PatchTST, and SparseTSF | Time-series forecasting experts · ETTh2, horizon 336 | MSE: 0.340 -> 0.4840 | Hurt |
| Apply MoE-F filtering over DLinear, PatchTST, and SparseTSF | Time-series forecasting experts · ETTh2, horizon 720 | MSE: 0.374 -> 0.3917 | Hurt |
| Add NIFTY SFT LoRA adapter (ablation) | Llama-2-7b-chat · NIFTY test split | F1: 0.22 -> 0.28 | Helped |
| Add ACL18 SFT LoRA adapter (ablation) | Llama-2-7b-chat · NIFTY test split | F1: 0.22 -> 0.20 | Hurt |
| Add BigData22 SFT LoRA adapter (ablation) | Llama-2-7b-chat · NIFTY test split | F1: 0.22 -> 0.29 | Helped |
Surprising findings
- Filtering over LLM experts yields large F1 gains even though no single expert is clearly superior.
- Replicated PatchTST results on ETTh2 show an outlier (2.6450 at horizon 336), indicating instability of some baselines.
Limitations
- Only LLM experts used in the financial experiment; no SDE-based experts combined.
- Ensemble performance is bounded by the quality of available experts; gains smaller with weaker expert sets (43% vs 52%).
- Theoretical guarantees rely on strong regularity and Gaussian-noise assumptions, and hyperparameters lambda and alpha are not tuned.
Experimental setup
Datasets: NIFTY ACL18 BigData22 CIKM ETTh1 ETTh2
Benchmarks: NIFTY ETTh1 ETTh2
Metrics: F1 accuracy precision recall MSE
Baselines: Llama-2-7b-chat Llama-2-70b-chat Llama-3-8B-Instruct Llama-3-70B-Instruct Mixtral-8x7B-Instruct-v0.1 DBRX-Instruct GPT-4o Informer Autoformer Pyraformer FEDformer FiLM
Builds on: Wonham-Shiryaev filter Gibbs posterior mixtures PAC-Bayes aggregation LoRA
Closest papers by meaning
- Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts (ICLR 2025 · similarity 0.88)
- FilterNet: Harnessing Frequency Filters for Time Series Forecasting (NeurIPS 2024 · similarity 0.86)
- Approaching Human-Level Forecasting with Language Models (NeurIPS 2024 · similarity 0.86)
- LangTime: A Language-Guided Unified Model for Time Series Forecasting with Proximal Policy Optimization (ICML 2025 · similarity 0.86)
- AutoTimes: Autoregressive Time Series Forecasters via Large Language Models (NeurIPS 2024 · similarity 0.86)
- Time-LLM: Time Series Forecasting by Reprogramming Large Language Models (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.