Research map / Large language models / Language, safety and interpretability
Seq vs Seq: An Open Suite of Paired Encoders and Decoders
Orion Weller, Kathryn Ricci, Marc Marone, Antoine Chaffin, Dawn Lawrie, Benjamin Van Durme
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the same recipe for encoder as ModernBERT with open data (Ettin-Enc) instead of ModernBERT | Ettin-Enc-150m · GLUE average (base size) | GLUE average: 88.4 -> 88.9 | Improved |
| Ettin decoder recipe versus Llama 3.2 1B | Ettin-Dec-1B · generative task average (1B) | average accuracy: 56.6 -> 59.0 | Improved |
| continue pre-training a decoder with MNTP for 50B tokens (encoder-from-decoder) instead of using a pure encoder (ablation) | Ettin-Dec-400m · MNLI (400M) | accuracy: 91.3 (encoder) vs 87.6 (enc-from-dec) | Hurt |
| add 50B tokens of MNTP continued training to a decoder for retrieval (ablation) | Ettin-Dec-400m · MS MARCO dev (400M) | nDCG@10: 39.93 -> 41.44 | Improved |
| pure encoder instead of encoder-from-decoder for retrieval (ablation) | Ettin-Enc-from-Dec-400m · MS MARCO dev (400M) | nDCG@10: 41.44 -> 42.24 | Improved |
| continue pre-training an encoder with CLM (decoder-from-encoder) instead of a pure decoder (ablation) | Ettin-Enc-1B · generative average (1B) | average accuracy: 59.0 (decoder) vs 52.2 (dec-from-enc) | Hurt |
| model scale from 68m to 1B for decoder-from-encoder (ablation) | decoder-from-encoder · generative tasks | average accuracy: 42.1 vs 41.8 at 68m; 52.2 vs 59.0 at 1B | Hurt |
| use an encoder in generative fashion on classification-like tasks (ARC) | Ettin-Enc-400m · ARC (400M) | accuracy: 33.6 -> 35.6 | Improved |
| continue-train an encoder with CLM on a classification-style task (MMLU CS) (ablation) | Ettin-Enc-1B · MMLU CS (1B) | accuracy: 27.0 -> 37.0 | Improved |
| continue-train an encoder with CLM on a generative task (GSM8k) (ablation) | Ettin-Enc-1B · GSM8k (1B) | accuracy: 32.0 -> 18.9 | Hurt |
| use non-filtered data instead of filtered DCLM and higher-quality sources in later phases (ablation) | Ettin training data · internal ablation | downstream performance | Hurt |
| model merging (not used by Ettin) | Ettin encoders · downstream applications | performance | Improved |
Closest papers by meaning
- A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation (CVPR 2025 · similarity 0.89)
- Reference Trustable Decoding: A Training-Free Augmentation Paradigm for Large Language Models (NeurIPS 2024 · similarity 0.88)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.88)
- MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding Methods (ICLR 2024 · similarity 0.88)
- Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe (NeurIPS 2024 · similarity 0.87)
- EmbedLLM: Learning Compact Representations of Large Language Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
