Research map / Large language models / Language, safety and interpretability

ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities

ICLR 2025 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Extends Llama3-70B to a 128K context via continued pretraining and a three-stage instruction tuning recipe, then compares its long-context and RAG performance against GPT-4-Turbo and other long-context LLMs, releasing weights, data and recipe.

Problem

Open-access LLMs lag proprietary models on real-world long-context understanding and RAG, and prior long-context models lack reproducible training recipes and are mainly evaluated on synthetic tasks; it is also unclear whether long-context LLMs or RAG is better for a given input length.

Approach

Continued pretraining of Llama3 base from 8K to 128K with upsampled long documents and a RoPE base of 150M, followed by three-stage instruction tuning (including synthetic 32K-128K NarrativeQA-based SFT data), paired with the E5-mistral long-context retriever and a study showing RAG with many retrieved chunks beats direct long-context input.

Key results

• Llama3-ChatQA-2-70B averages 41.04 on InfiniteBench ultra-long (>100K) tasks, above GPT-4-Turbo (33.16) and Llama3.1-70B-Instruct (39.81).
• On ChatRAG Bench (4K), Llama3-ChatQA-2-70B averages 56.30, above GPT-4-Turbo (54.72) and Qwen2-72B-Instruct (54.06).
• On 32K tasks, RAG with many retrieved tokens (at least 12K) marginally outperforms direct long context, and for >100K tasks RAG with top-30 chunks gives 64.55 vs 64.29 long context for ChatQA-2-70B.

Measured effects reported in the paper

Surprising findings

• Llama3.1-8B and 70B Instruct fail the NIAH test with the harder 'San Francisco' needle, though they pass passkey retrieval.
• Three-stage instruction tuning substantially outperforms all-in-one single-stage training on RAG (52.50 vs 48.55 short).

Limitations

• Smaller continued pretraining corpus and SFT data than Qwen2 and Llama3.1, leading to lower 32K scores.
• Weak summarization performance due to lack of summarization data in SFT.
• Extending to 128K slightly reduces short-context performance relative to ChatRAG-specialized 4K model.

Experimental setup

Datasets: SlimPajama LongAlpaca12k OpenOrca Long-Data-Collections NarrativeQA ChatRAG Bench training blend

Benchmarks: InfiniteBench (En.Sum, En.QA, En.MC, En.Dia) SCROLLS and LongBench (QMSum, Qasper, QuALITY, MuSiQue, HotpotQA, MultiFieldQA) ChatRAG Bench Needle In A Haystack GSM8K MMLU HumanEval MT-Bench

Metrics: F1 ROUGE exact match accuracy

Baselines: GPT-4-Turbo-2024-04-09 Qwen2-72B-Instruct Llama3.1-70B-Instruct Llama3.1-8B-Instruct Llama-3-70B-Instruct-Gradient-262k Claude 2 Llama3-ChatQA-1.5-70B

Builds on: ChatQA 1.5 Llama 3 E5-mistral YaRN InfiniteBench

Closest papers by meaning

• Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG (ICLR 2025 · similarity 0.91)
• LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs – No Silver Bullet for LC or RAG Routing (ICML 2025 · similarity 0.90)
• LooGLE v2: Are LLMs Ready for Real World Long Dependency Challenges? (NeurIPS 2025 · similarity 0.89)
• LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (ICML 2024 · similarity 0.89)
• One Token Can Help! Learning Scalable and Pluggable Virtual Tokens for Retrieval-Augmented Large Language Models (AAAI 2025 · similarity 0.89)
• RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decoding (ICML 2025 · similarity 0.89)