Research map / Vision-language and multimodal / Visual question answering
cPAPERS: A Dataset of Situated and Multimodal Interactive Conversations in Scientific Papers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces cPAPERS, a dataset of 5,030 OpenReview review/rebuttal question-answer pairs grounded in equations, tables, and figures of arXiv papers, and benchmarks zero-shot Llama-2-70B and QLoRA-tuned Llama-2-7B baselines.
Problem
Conversational question answering over scientific papers requires grounding in equations, tables, and figures, which prior QA and multimodal datasets do not cover conversationally.
Approach
Mines reviewer questions and author rebuttals from OpenReview with an LLM and regex filtering, then links them to arXiv LaTeX sources to attach captions, context paragraphs, and references, with neighboring-equation/table grounding to handle version mismatches.
Key results
- Zero-shot Llama-2-70B with neighboring equations gives up to 2x ROUGE/METEOR/BERTScore gains over using all equations, and question-only input is best on ROUGE-2 in equation split.
- Fine-tuned Llama-2-7B with Q+Equation achieves ROUGE-1 0.317 and BERTScore 0.861 on cPAPERS-EQNS, but the ANOVA test shows the difference is not statistically significant.
- Fine-tuning on tables and figures: question-only input performs best across most metrics (e.g., ROUGE-1 0.329 on FIGS).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use neighboring equations rather than all equations (ablation) | Llama-2-70B zero-shot on cPAPERS-EQNS · cPAPERS-EQNS test | ROUGE-1: 0.170 -> 0.190 | Helped |
| Add neighboring equations to the question (ablation) | Llama-2-70B zero-shot on cPAPERS-EQNS · cPAPERS-EQNS test | ROUGE-1: 0.194 -> 0.190 | Hurt |
| Add all equations to the question (ablation) | Llama-2-70B zero-shot on cPAPERS-EQNS · cPAPERS-EQNS test | ROUGE-1: 0.194 -> 0.170 | Hurt |
| Add neighboring table references to the question (ablation) | Llama-2-70B zero-shot on cPAPERS-TBLS · cPAPERS-TBLS test | ROUGE-1: 0.192 -> 0.207 | Helped |
| Use neighboring rather than all table references (ablation) | Llama-2-70B zero-shot on cPAPERS-TBLS · cPAPERS-TBLS test | ROUGE-1: 0.186 -> 0.207 | Helped |
| Add context to the question (ablation) | Llama-2-70B zero-shot on cPAPERS-FIGS · cPAPERS-FIGS test | ROUGE-1: 0.185 -> 0.208 | Helped |
| Add the equation to the question (ablation) | Llama-2-7B QLoRA on cPAPERS-EQNS · cPAPERS-EQNS test | ROUGE-1: 0.309 -> 0.317 | No clear effect |
| Add context to the question (ablation) | Llama-2-7B QLoRA on cPAPERS-EQNS · cPAPERS-EQNS test | ROUGE-1: 0.309 -> 0.297 | Hurt |
| Add references to the question (ablation) | Llama-2-7B QLoRA on cPAPERS-EQNS · cPAPERS-EQNS test | ROUGE-1: 0.309 -> 0.283 | Hurt |
| Add the table to the question (ablation) | Llama-2-7B QLoRA on cPAPERS-TBLS · cPAPERS-TBLS test | ROUGE-1: 0.315 -> 0.293 | Hurt |
| Add context to the question (ablation) | Llama-2-7B QLoRA on cPAPERS-TBLS · cPAPERS-TBLS test | ROUGE-1: 0.315 -> 0.294 | Hurt |
| Add references to the question (ablation) | Llama-2-7B QLoRA on cPAPERS-TBLS · cPAPERS-TBLS test | ROUGE-1: 0.315 -> 0.292 | Hurt |
Surprising findings
- Question-only input often matches or beats inputs with context or references after fine-tuning, which the authors attribute to reviewers using figures and tables to supplement questions rather than to answer them.
- Providing all equations or tables from the paper roughly halves the zero-shot scores compared with a neighboring subset.
Limitations
- Mismatches between arXiv LaTeX versions and the OpenReview submission cause misaligned equations, tables, and figures.
- Ordinal references (Equation 3, Table 2) cannot be reliably mapped to compiled document positions, so neighboring-subset grounding is approximate.
- Evaluation relies on n-gram and embedding overlap metrics only, with no human or LLM judging of answer quality.
Experimental setup
Datasets: cPAPERS OpenReview reviews and rebuttals arXiv LaTeX sources
Benchmarks: cPAPERS-EQNS cPAPERS-TBLS cPAPERS-FIGS
Metrics: ROUGE-1 ROUGE-2 ROUGE-L METEOR BERTScore
Baselines: Question-only input Llama-2-70B zero-shot
Builds on: QLoRA Llama-2
Closest papers by meaning
- SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers (NeurIPS 2024 · similarity 0.87)
- Are Large Language Models Ready for Multi-Turn Tabular Data Analysis? (ICML 2025 · similarity 0.84)
- PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistencies (ICLR 2026 · similarity 0.83)
- MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models (AAAI 2026 · similarity 0.83)
- ChatterBox: Multimodal Referring and Grounding with Chain-of-Questions (AAAI 2025 · similarity 0.82)
- AAAR-1.0: Assessing AI’s Potential to Assist Research (ICML 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.