Research map / Large language models / Instruction and fine-tuning
PRAGWORLD: A Benchmark Evaluating LLMs’ Local World Model Under Minimal Linguistic Alterations and Conversational Dynamics
Sachin Vashistha, Aryan Bibhuti, Atharva Naik, Martin Tutek, Somak Aditya
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune on the synthetic split | Llama-3.2-3B-Instruct · PragWorld (manual) | robust accuracy: 20.41 -> 48.98 | Improved |
| fine-tune on the synthetic split | Qwen2.5-1.5B-Instruct · PragWorld (manual) | robust accuracy: 22.45 -> 47.96 | Improved |
| fine-tune on the synthetic split | Phi-3.5-mini-instruct · PragWorld (manual) | robust accuracy: 48.98 -> 52.04 | Improved |
| fine-tune on the synthetic split | Qwen2.5-0.5B-Instruct · PragWorld (manual) | robust accuracy: 19.39 -> 22.45 | Improved |
| fine-tune on the synthetic split | Llama-3.2-1B-Instruct · PragWorld entity tracking probe | total accuracy: 52.19 -> 71.97 | Improved |
| fine-tune on the synthetic split | models of 7-8B · PragWorld entity tracking probe | total accuracy: 85.16 -> 87.91 (Llama-3.1-8B) | Improved |
| fine-tune on the synthetic split | PragWorld manual split · PragWorld (manual) | Yes/No accuracy gap | No clear effect |
| add Harmful Layer Suppression (L2 penalty on MLP output of harmful layers) during fine-tuning | Phi-3.5-mini-instruct · PragWorld MLP zero-out ablation | accuracy | Improved |
| add Useful Layer Amplification (classification heads on useful layers) during fine-tuning | Qwen2.5-7B-Instruct · PragWorld MLP zero-out ablation | accuracy | Improved |
| zero out MLP in harmful layers (5, 6, 7, 11, 13, 17, 31) (ablation) | Phi-3.5-mini-instruct · PragWorld | accuracy | Improved |
| zero out MLP in useful layers (2, 9, 16) (ablation) | Phi-3.5-mini-instruct · PragWorld | accuracy | Hurt |
Closest papers by meaning
- LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset (ICLR 2024 · similarity 0.89)
- Sparsity-Guided Holistic Explanation for LLMs with Interpretable Inference-Time Intervention (AAAI 2024 · similarity 0.87)
- Mutable Transcripts: Mitigating Context Pollution through Editable Conversation State (NeurIPS 2026 · similarity 0.87)
- Exploring Conversational Adaptability: Assessing the Proficiency of Large Language Models in Dynamic Alignment with Updated User Intent (AAAI 2025 · similarity 0.87)
- Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Models (NeurIPS 2024 · similarity 0.87)
- Potemkin Understanding in Large Language Models (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
