Research map / Large language models / Retrieval and knowledge
ChatQA: Surpassing GPT-4 on Conversational QA and RAG
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ChatQA, a two-stage instruction-tuned family of models (up to Llama3-70B) plus a fine-tuned conversational retriever and the ChatRAG Bench, matching or beating GPT-4 on conversational QA and RAG.
Problem
Building open models that match proprietary GPT-4 on conversational question answering and retrieval-augmented generation, including follow-up questions, long-context retrieval, tables, arithmetic, and unanswerable questions.
Approach
A two-stage instruction tuning recipe (SFT followed by context-enhanced instruction tuning with conversational and single-turn QA data plus a small set of unanswerable samples), together with a single-turn retriever fine-tuned on multi-turn query-context pairs that matches LLM-based query rewriting at lower cost.
Key results
- ChatQA-1.0-70B averages 54.14 on ChatRAG Bench, slightly above GPT-4-Turbo-2024-04-09 (54.03) and GPT-4-0613 (53.90).
- Llama3-ChatQA-1.5-70B averages 58.25 (57.14 excluding HDial), surpassing GPT-4-Turbo by about 4.4% per the abstract.
- Removing stage-2 context-enhanced tuning drops ChatQA-1.0-70B average from 54.14 to 43.22; fine-tuned Dragon retriever matches query rewriting with GPT-3.5-Turbo on average top-5 recall (52.72 vs 54.46 top-1 for rewrite).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply two-stage instruction tuning (ablation) | ChatQA-1.0-70B · ChatRAG Bench | average score: 43.22 -> 54.14 | Helped |
| include stage-1 SFT (ablation) | ChatQA-1.0-70B · ChatRAG Bench | average score: 52.18 -> 54.08 | Helped |
| include single-turn QA datasets in stage-2 (ablation) | ChatQA-1.0-70B · ChatRAG Bench | average score: 52.25 -> 54.08 | Helped |
| include HumanAnnotatedConvQA in stage-2 (ablation) | ChatQA-1.0-70B · ChatRAG Bench | average score: 48.97 -> 54.08 | Helped |
| replace HumanAnnotatedConvQA with SyntheticConvQA (ablation) | ChatQA-1.0-70B · ChatRAG Bench | average score: 54.14 -> 54.08 | No clear effect |
| fine-tune Dragon on conversational query-context pairs instead of using query rewriting | Dragon retriever · Five conversational retrieval datasets | average top-1 recall: 54.46 -> 52.72; 80.13 -> 80.67 | Mixed |
| fine-tune E5-unsupervised on conversational query-context pairs | E5-unsupervised · Five conversational retrieval datasets | average top-1 recall: 33.23 -> 47.79 | Helped |
| use the human-annotated conversational QA data instead of SyntheticConvQA (ablation) | ChatQA-1.0-70B · QuAC and DoQA unanswerable evaluation | average accuracy: 69.84 -> 77.25 | Helped |
| use 1.5k unanswerable training samples instead of 1k (ablation) | ChatQA-1.0-70B · QuAC and DoQA unanswerable evaluation | average accuracy: 76.88 -> 77.25 | Helped |
| increase unanswerable training samples from 1.5k to 2.5k (ablation) | ChatQA-1.0-70B · QuAC and DoQA unanswerable evaluation | average accuracy: 77.25 -> 75.87 | Hurt |
| use top-5 retrieved chunks for stage-2 training context (ablation) | ChatQA-1.0-70B · Retrieval datasets vs. non-retrieval datasets | score: 42.33 -> 42.91; 65.96 -> 65.16 | Mixed |
| replace Dragon + Fine-tune with the original Dragon retriever (ablation) | ChatQA-1.0-70B · Retrieval QA datasets | average score: 42.31 -> 40.50 | Hurt |
Surprising findings
- Fine-tuning a single-turn retriever on conversational data performs on par with GPT-3.5-Turbo query rewriting, and fine-tuning E5-unsupervised improves recall by more than 15 points over rewriting.
- Replacing human-annotated data with GPT-3.5-Turbo synthetic data yields comparable average scores, and using top-5 retrieved chunks in stage-2 training helps retrieval datasets but hurts non-retrieval ones.
Limitations
- Still a gap to GPT-4 on tabular reasoning (SQA human win rate) and on unanswerable cases (around 3.5% behind GPT-4-0613).
- Llama3-ChatQA-1.5 variants answer unanswerable questions from parametric knowledge, lowering accuracy on unanswerable cases.
- Human evaluation covers only ChatQA-1.0-70B vs GPT-4-0613.
Experimental setup
Datasets: Soda ELI5 FLAN chain-of-thought datasets Self-Instruct Unnatural Instructions OpenAssistant Dolly HumanAnnotatedConvQA SyntheticConvQA DROP NarrativeQA
Benchmarks: ChatRAG Bench Natural Questions TriviaQA HotpotQA
Metrics: F1 exact match top-1 recall top-5 recall accuracy on answerable and unanswerable samples human A/B win rate
Baselines: Llama2-Chat Llama3-Instruct Command R+ GPT-3.5-Turbo-0613 GPT-4-0613 GPT-4-Turbo-2024-04-09 E5-unsupervised Dragon
Builds on: Dragon Llama 2 Llama 3 RA-DIT
Closest papers by meaning
- RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs (NeurIPS 2024 · similarity 0.89)
- Cooperative Retrieval-Augmented Generation for Question Answering: Mutual Information Exchange and Ranking by Contrasting Layers (NeurIPS 2025 · similarity 0.88)
- Automated Evaluation of Retrieval-Augmented Language Models with Task-Specific Exam Generation (ICML 2024 · similarity 0.87)
- ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities (ICLR 2025 · similarity 0.87)
- RAG-IGBench: Innovative Evaluation for RAG-based Interleaved Generation in Open-domain Question Answering (NeurIPS 2025 · similarity 0.87)
- InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.