Research map / Large language models / Language, safety and interpretability
Teaching Metric Distance to Discrete Autoregressive Language Models
Jiwan Chung, Saejin Kim, Yongrae Jo, Jaewoo Park, Dongjun Min, Youngjae Yu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add place value weighting (ablation) | DIST2Loss (Llama-3.2-1B) · meta linear regression | MAE: 0.098 -> 0.092 | Improved |
| add contrastive loss (ablation) | DIST2Loss (Llama-3.2-1B) · meta linear regression | MAE: 0.099 -> 0.092 | Improved |
| add distance-aware target (vs label smoothing 0.1) (ablation) | DIST2Loss (Llama-3.2-1B) · meta linear regression | MAE: 0.099 -> 0.092 | Improved |
| use a random distance metric instead of Euclidean (ablation) | DIST2Loss · reward modeling | accuracy (%): 85.3 -> 76.0 | Hurt |
| add DIST2Loss to SFT | Llama-3.1-8B · RewardBench average | accuracy (%): 75.3 -> 85.3 | Improved |
| add DIST2Loss to SFT | Phi-3.5-vision · RefCOCO+ testB | accuracy (%): 78.7 -> 81.4 | Improved |
| add vocabulary-based soft target instead of DIST2Loss | Phi-3.5-vision · RefCOCO+ testA | accuracy (%): 91.6 -> 90.6 | Hurt |
| add DIST2Loss to SFT | LLaRA · VIMABench L2 (1K data) | accuracy (%): 46.2 -> 51.5 | Improved |
| add DIST2Loss to SFT | LlamaGen 111M · ImageNet 256 (50 epochs) | FID: 10.03 -> 9.41 | Improved |
| add DIST2Loss to SFT | LlamaGen 343M · ImageNet 256 (300 epochs) | FID: 3.08 -> 3.04 | Improved |
| set loss weight alpha to 0.1 (vs 1.0) (ablation) | DIST2Loss · reward modeling | accuracy (%): 77.3 -> 85.3 | Improved |
| set loss weight alpha to 0.001 (vs 0.1) (ablation) | DIST2Loss · reward modeling | accuracy (%): 85.3 -> 75.6 | Hurt |
Closest papers by meaning
- Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Models (NeurIPS 2024 · similarity 0.87)
- Where MLLMs Attend and What They Rely On: Explaining Autoregressive Token Generation (CVPR 2026 · similarity 0.87)
- Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flows (NeurIPS 2025 · similarity 0.86)
- Beyond Next-Token Alignment: Distilling Multimodal Large Language Models via Token Interactions (ICML 2026 · similarity 0.86)
- Grounding Multimodal Large Language Models in Actions (NeurIPS 2024 · similarity 0.86)
- Hierarchical Autoregressive Transformers: Combining Byte- and Word-Level Processing for Robust, Adaptable Language Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
