AI Paper Atlas

Research map / Learning theory and algorithms / Prediction and decision losses

Geometric and Stochastic Analysis of Discontinuities in Sparse Mixture-of-Experts

Tho Tran Huu, Huu-Tuan Nguyen, Thien-Hai Nguyen, Nhat-Tri Ho, Viet-Hoang Tran, Tho Quan, Tan Minh Nguyen

ICML 2026 · Learning theory and algorithms › Prediction and decision losses · approach: Determinantal point process inference · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add l-inf epsilon local smoothing of gating logits near top-k boundariesSwitch Transformer SMoE · WikiText-103 testperplexity: 35.52 -> 34.35Improved
add l-inf epsilon local smoothingSwitch Transformer SMoE · Attacked WikiText-103 testperplexity: 44.18 -> 42.85Improved
add smoothing instead of ReMoE continuous routingSwitch Transformer SMoE · WikiText-103 testperplexity: 35.35 -> 34.35Improved
add smoothingSwitch Transformer SMoE · EnWiki-8bits per character: 1.153 -> 1.122Improved
add smoothing (k=2)BERT-large with one MoE layer · GLUE average (5 tasks)average score: 81.17 -> 81.65Improved
add smoothing (k=4)BERT-large with one MoE layer · GLUE average (5 tasks)average score: 81.14 -> 81.73Improved
add smoothing (k=2)BERT-large with one MoE layer · GLUE QNLIaccuracy: 92.56 -> 92.40No clear effect
add smoothing to GMoEGMoE ViT-S/16 · DomainBed averageaccuracy (%): 66.84 -> 67.4Improved
add smoothing to GMoEGMoE ViT-S/16 · DomainBed PACSaccuracy (%): 87.7 -> 87.6No clear effect
anneal smoothing away during training and use hard top-2 at inference (ablation)SmoothSMoE · WikiText-103 testperplexity: 35.52 -> 34.62Improved
keep smoothing at inference (k=2.5) versus annealed hard top-2 (ablation)SmoothSMoE · WikiText-103 testperplexity: 34.62 -> 34.35Improved
add smoothing (k=2.5) to upcycled VLM5.6B ViT-based VLM, 4 experts · Vision-language average over 10 benchmarksaverage score: 54.53 -> 55.07Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy