Research map / Architectures and efficiency / Pruning and efficient tuning
MoSE: Hierarchical Self-Distillation Enhances Early Layer Embeddings
Andrea Gurioli, Federico Pennino, Joao Monteiro, Maurizio Gabbrielli
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add multi-layer self-distillation loss at fine-tuning (vs single-exit fine-tuning) (ablation) | MoSE exit heads · CodeSearchNet | Recall@1: up to +4.36% averaged Recall@1 at layer 9 | Improved |
| exit at layer 4 instead of layer 36 | MoSE · CodeSearchNet text-to-code | avg MRR: 6.4% absolute drop for about 90% fewer FLOPs | Hurt |
| use shallow layers for code-to-code retrieval | MoSE · POJ104 | mAP | Improved |
| use layer 18 for text-to-code retrieval | MoSE · CodeSearchNet | MRR | Improved |
| replace NSP with In-Context Classification in pretraining (ablation) | MoSE-30M · CoIR CSN-CCR | NDCG@10: 6.2 -> 10.1 | Improved |
| replace NSP with In-Context Classification in pretraining (ablation) | MoSE-30M · CoIR CoSQA, CT-DL, CT-Contest | NDCG@10: 31.0 -> 32.0 on CT-DL; 12.0 -> 12.1 on CT-Contest; 0.4 on CoSQA | No clear effect |
| go deeper than layer 18 on code clone detection | MoSE · BigCloneBench | F1: 94.2 (L18) vs 94.1 (L36) | No clear effect |
| use very early exits (L4, L9) for clone detection | MoSE · BigCloneBench | precision: 89.8 (L4), 90.4 (L9) vs 91.7 (L36) | Hurt |
Closest papers by meaning
- SelfCodeAlign: Self-Alignment for Code Generation (NeurIPS 2024 · similarity 0.88)
- CODE REPRESENTATION LEARNING AT SCALE (ICLR 2024 · similarity 0.88)
- Towards A Generalist Code Embedding Model Based On Massive Data Synthesis (NeurIPS 2025 · similarity 0.87)
- AlchemistCoder: Harmonizing and Eliciting Code Capability by Hindsight Tuning on Multi-source Data (NeurIPS 2024 · similarity 0.86)
- Scaling Embedding Layers in Language Models (NeurIPS 2025 · similarity 0.86)
- Repoformer: Selective Retrieval for Repository-Level Code Completion (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
