Research map / Large language models / Agents, code and math
Evaluating Whether LLMs Can Reliably Connect the DOTs?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a ~9.1K-instance multi-domain narrative infilling benchmark from Wikipedia, ROCStories, SIND and CNN/DailyMail and evaluates 20 open instruction-tuned LLMs (1.5B–70B) under varying prompt specificity and reasoning strategies using automatic metrics and GPT-4.1-mini as judge.
Problem
Text infilling is used as a pre-training objective, but LLMs' ability to reconstruct missing spans in real-world narratives while staying consistent with local and global context is underexplored.
Approach
A multi-domain narrative infilling benchmark created by masking 1–3 consecutive sentences at opening, middle, or closing positions, paired with a five-dimension qualitative rubric judged by an LLM and validated against humans, and a systematic study of instruction specificity (TELeR levels) and reasoning paradigms.
Key results
- Gemma-2-2B achieves the highest qualitative score (4.02/5), above DeepSeek-Qwen-32B (3.77/5) and LLaMA-3.3-70B (3.71/5).
- Chain-of-thought reasoning improves average qualitative score only marginally (3.822 vs 3.80 at moderate specificity, +0.6%).
- Automatic metrics correlate weakly with qualitative dimensions (max Kendall tau 0.141).
Surprising findings
- A 2B model (Gemma-2-2B) outperforms models over ten times larger on qualitative narrative infilling scores.
- Counterfactual reasoning degrades narrative quality, and very high specificity with few-shot examples lowers scores for many models.
Limitations
- English-only and instruction-tuned checkpoints only.
- Contiguous span masking does not cover non-contiguous deletions or stylistic revisions.
- LLM judge may not fully align with human judgment; four rubric dimensions near ceiling.
Experimental setup
Datasets: ROCStories SIND CNN/DailyMail Wikipedia
Benchmarks: Narrative Infilling benchmark (9,142 instances)
Metrics: BERTScore ROUGE-1 chrF METEOR Sem-F1 Fluency Context Faithfulness Bidirectional Coherence Narrative Consistency Informativeness
Baselines: Gemma-2-2B DeepSeek-Qwen-32B LLaMA-3.3-70B
Builds on: TELeR taxonomy chain-of-thought prompting
Closest papers by meaning
- NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens (ICLR 2025 · similarity 0.86)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.84)
- Make Your LLM Fully Utilize the Context (NeurIPS 2024 · similarity 0.84)
- Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.84)
- Beyond Surface Structure: A Causal Assessment of LLMs' Comprehension ability (ICLR 2025 · similarity 0.84)
- OmniToM: Benchmarking Theory of Mind in LLMs via Explicit Belief Modeling (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.