Research map / Architectures and efficiency / Attention and transformers
ConceptMoE: Adaptive Token-to-Concept Compression for Implicit Compute Allocation
Zihao Huang, Jundong Zhou, Xingwei Qu, Qiyang Min, Ge Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ConceptMoE instead of MoE | MoE-A0.5B-12B · OpenBench Easy | All score: 35.6 -> 36.4 | Improved |
| use ConceptMoE instead of MoE | MoE-A1B-24B · OpenBench Easy | All score: 50.0 -> 50.9 | Improved |
| use ConceptMoE instead of MoE | MoE-A2.5B-60B · Long context | All score: 49.4 -> 51.7 | Improved |
| use ConceptMoE instead of MoE | MoE-A2.5B-60B · Vision-language | All score: 53.6 -> 54.2 | Improved |
| convert MoE to ConceptMoE-top15 in continual training | MoE-A2.5B-90B · Open Benchmark | Overall score: 40.9 -> 41.3 | No clear effect |
| add layer loops (ConceptMoE-top11-loop8) in CT conversion | MoE-A2.5B-90B · Open Benchmark | Overall score: 40.9 -> 46.4 | Improved |
| train ConceptMoE-top11-loop8 from scratch rather than CT conversion | ConceptMoE-top11-loop8 · Open Benchmark | Overall score: 46.4 -> 47.3 | Improved |
| add learnable dynamic chunking instead of fixed chunking (ablation) | ConceptMoE · OpenBench Easy All (A0.5B-12B) | All score: 34.2 -> 36.4 | Improved |
| use cosine router instead of linear router (ablation) | ConceptMoE · OpenBench Easy All (A0.5B-12B) | All score: 34.4 -> 36.4 | Improved |
| use cosine router instead of linear router (ablation) | ConceptMoE · A0.5B-12B | training loss: 0.003 lower loss for linear | Hurt |
| add joint decoding of concepts and tokens (ablation) | ConceptMoE · OpenBench Easy All (A0.5B-12B) | All score: 35.1 -> 36.4 | Improved |
| add Bernoulli boundary noise tau=4 (ablation) | ConceptMoE · OpenBench Easy All (R=1.5) | All score: 28.9 -> 30.3 | Improved |
Closest papers by meaning
- ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference (NeurIPS 2025 · similarity 0.88)
- Task-Related Token Compression in Multimodal Large Language Models from an Explainability Perspective (ICLR 2026 · similarity 0.88)
- Filter, Correlate, Compress: Training-Free Token Reduction for MLLM Acceleration (AAAI 2026 · similarity 0.88)
- EvoComp: Learning Visual Token Compression for Multimodal Large Language Models via Semantic-Guided Evolutionary Labeling (CVPR 2026 · similarity 0.87)
- Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers (NeurIPS 2025 · similarity 0.87)
- Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
