Research map / Large language models / Language, safety and interpretability
Two Heads Are Better than One: Distilling Large Language Model Features into Small Models with Feature Decomposition and Mixture
Tianhao Fu, Xinxin Xu, Weichen Xu, Jue Chen, Ruilong Ren, Bowen Deng, Xinyu Zhao, Jian Cao, Xixin Cao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add task decomposition to layer decomposition (ablation) | CMM FD(layer) · FU | EPnL (1e3): 12.54 -> 17.25 | Improved |
| add volatility-type data decomposition (ablation) | CMM FD(task) · FU | EPnL (1e3): 17.25 -> 22.41 | Improved |
| add Hajek-MoE fusion of experts (ablation) | CMM FD(type) · FU | EPnL (1e3): 22.41 -> 31.39 | Improved |
| add Hajek-MoE fusion of experts (ablation) | CMM FD(type) · FU | MAP (unit): 45 -> 32 | Improved |
| distill LLM features with orthogonal decomposition instead of ReviewKD | small student · FU | EPnL (1e3): 10.52 -> 31.39 | Improved |
| use CMM instead of X-MoE | MoE fusion · FU | EPnL (1e3): 19.63 -> 31.39 | Improved |
| distill LLM into CMM small models | LLM-Base · FU | EPnL (1e3): 30.47 -> 31.39 | Improved |
| distill LLM into CMM small models | LLM-Base · FU | latency (s): 1.9 -> 0.3 | Improved |
| use CMM instead of IMM | IMM · RB | EPnL (1e3): 16.46 -> 22.69 | Improved |
| use CMM instead of IMM | IMM · FU | EPnL (1e3): 28.1 -> 31.39 | Improved |
| use CMM instead of IMM | IMM · AG | EPnL (1e3): -14.5 -> -8.95 | Improved |
| use CMM instead of IMM under extreme market | IMM · simulated flash crash and reversal | EPnL (1e3): 7.80 -> 10.50 | Improved |
Closest papers by meaning
- Market-GAN: Adding Control to Financial Market Data Generation with Semantic Context (AAAI 2024 · similarity 0.86)
- MarS: a Financial Market Simulation Engine Powered by Generative Foundation Model (ICLR 2025 · similarity 0.86)
- FinBen: A Holistic Financial Benchmark for Large Language Models (NeurIPS 2024 · similarity 0.86)
- FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Making (NeurIPS 2024 · similarity 0.86)
- Towards Financial World Modeling (NeurIPS 2026 · similarity 0.86)
- Breaking the Gradient Barrier: Unveiling Large Language Models for Strategic Classification (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
