Research map / Large language models / Language, safety and interpretability
LooGLE v2: Are LLMs Ready for Real World Long Dependency Challenges?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LooGLE v2, a benchmark of 1,934 closed-form QA instances over real-world long documents (16K to 2M tokens) across law, finance, game, and code, and evaluates 10 LLMs, with the best scoring 59.2%.
Problem
Existing long-context benchmarks focus on shallow retrieval and document QA and rarely test long-dependency reasoning over real-world, domain-specific long texts.
Approach
A scalable, automated pipeline collects real-world long texts (US court cases, SEC 10-K reports, CS2 and Crafter game logs, GitHub repositories) and builds 10 domain-specific long-dependency tasks with closed-form answers, so the benchmark can be refreshed with new data to limit contamination.
Key results
- GPT-4.1 (1M context) is the best model with an average score of 59.2% across the 10 tasks.
- GPT-4.1 scores 69.35% on Legal Article Extraction, the only model with a decent score on that task.
- Performance of API models drops as input length grows, with a noticeable decline beyond 128K tokens for GPT-4.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add chain-of-thought prompting (ablation) | LLaMA-3.1-8B-Instruct · LooGLE v2 average | score: 24.16 -> 27.94 | Helped |
| add chain-of-thought prompting (ablation) | Qwen2.5-7B-Instruct-1M · LooGLE v2 average | score: 28.97 -> 28.91 | No clear effect |
| add chain-of-thought prompting (ablation) | GLM-4-9b-chat · LooGLE v2 average | score: 25.81 -> 26.54 | Helped |
| add chain-of-thought prompting (ablation) | Mistral-7B-v0.2-Instruct · LooGLE v2 average | score: 11.88 -> 8.57 | Hurt |
| add chain-of-thought prompting (ablation) | Qwen2.5-7B-Instruct-1M · Metric Calculation | score: 84.00 -> 82.00 | Hurt |
| add chain-of-thought prompting (ablation) | LLaMA-3.1-8B-Instruct · User Behavior Analysis | score: 15.84 -> 29.07 | Helped |
| add chain-of-thought prompting (ablation) | Mistral-7B-v0.2-Instruct · Call Graph Analysis | score: 23.94 -> 10.99 | Hurt |
| use retrieval-augmented generation | selected models on LooGLE v2 · LooGLE v2 | score | Hurt |
| retain more intermediate context (ablation) | evaluated models on long inputs · LooGLE v2 varying input lengths | score | Helped |
| increase function call-chain depth | models on Code call graph analysis · Call Graph Analysis | score | Hurt |
Surprising findings
- GPT-4.1 with a 1M-token window underperforms the smaller-window GPT-o3-mini (200K) on Call Graph Analysis and Trend Analysis.
- RAG generally degrades performance compared with feeding the full long context.
Limitations
- Covers only four domains and a subset of long-dependency tasks, leaving many real-world scenarios unexplored.
- Context lengths and average lengths vary across tasks, which may confound cross-task comparisons.
Experimental setup
Datasets: US court cases from CourtListener and Westlaw SEC EDGAR 10-K annual reports CS2 match records from HLTV Crafter LLM-agent trajectories GitHub Python repositories
Benchmarks: LooGLE v2
Metrics: accuracy Jaccard similarity relative error within 5%
Baselines: GPT-4.1 GPT-o3-mini DeepSeek-R1 DeepSeek-V3 Qwen2.5-7B-Instruct-1M Llama-3.1-8B-Instruct GLM-4-9B-Chat Mistral-7B-Instruct-v0.2 Phi-3-Medium-128K-Instruct Yarn-Mistral-7b-128k
Builds on: LooGLE LongBench v2 Needle-In-A-Haystack
Closest papers by meaning
- LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (ICML 2024 · similarity 0.89)
- ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities (ICLR 2025 · similarity 0.89)
- When LLMs Meet Cunning Texts: A Fallacy Understanding Benchmark for Large Language Models (NeurIPS 2024 · similarity 0.88)
- LLM Augmented LLMs: Expanding Capabilities through Composition (ICLR 2024 · similarity 0.88)
- HELMET: How to Evaluate Long-context Models Effectively and Thoroughly (ICLR 2025 · similarity 0.88)
- MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.