Research map / Large language models / Retrieval and knowledge
RetLLM-E: Retrieval-Prompt Strategy for Question-Answering on Student Discussion Forums
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RetLLM-E, a retrieval-augmented prompting pipeline that feeds Q&A-pair and course-document retrieval context plus an expert-style prompt into LLaMA-2-13B-chat to answer student forum questions in a DATA 100 course.
Problem
General-purpose LLMs answer course-specific student forum questions poorly, lacking institution- and course-specific context, and tend to hallucinate or produce irrelevant answers.
Approach
Combine two retrieval sources (historical TA Q&A pairs and filtered course documents, excluding homework solutions) with an expert-persona, tone, and abstain-if-unsure prompt structure, so a locally hosted LLM generates course-grounded answers.
Key results
- RetLLM-E achieved ROUGE-1 0.193 and BERTScore 0.838, versus 0.136 and 0.806 for zero-shot and 0.165 and 0.830 for no-retrieval.
- Human factuality score for RetLLM-E was 2.04 versus 1.92 (zero-shot) and 1.86 (no-retrieval) on a 1-3 scale.
- Relevance was 2.69 for RetLLM-E versus 2.33 for zero-shot, with no-retrieval at 2.68.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Q&A and course-document retrieval context to the engineered LLM prompt (ablation) | no-retrieval · DATA 100 stand-alone questions | Factuality: 1.860±0.80 -> 2.040±0.80 | Helped |
| Add Q&A and course-document retrieval context to the engineered LLM prompt (ablation) | no-retrieval · DATA 100 stand-alone questions | Relevance: 2.680±0.60 -> 2.690±0.66 | Helped |
| Add Q&A and course-document retrieval context to the engineered LLM prompt (ablation) | no-retrieval · DATA 100 stand-alone questions | Readability: 2.800±0.43 -> 2.770±0.49 | Hurt |
| Add Q&A and course-document retrieval context to the engineered LLM prompt (ablation) | no-retrieval · DATA 100 stand-alone questions | Tone: 2.820±0.41 -> 2.780±0.48 | Hurt |
| Add Q&A and course-document retrieval context to the engineered LLM prompt (ablation) | no-retrieval · DATA 100 stand-alone questions | ROUGE 1: 0.165 -> 0.193 | Helped |
| Add Q&A and course-document retrieval context to the engineered LLM prompt (ablation) | no-retrieval · DATA 100 stand-alone questions | ROUGE 2: 0.038 -> 0.045 | Helped |
| Add Q&A and course-document retrieval context to the engineered LLM prompt (ablation) | no-retrieval · DATA 100 stand-alone questions | BERTScore: 0.830 -> 0.838 | Helped |
| Add expert, task, and tone preconditioning to a student-question-only prompt (ablation) | zero-shot · DATA 100 stand-alone questions | Relevance: 2.330±0.78 -> 2.680±0.60 | Helped |
| Add expert, task, and tone preconditioning to a student-question-only prompt (ablation) | zero-shot · DATA 100 stand-alone questions | Readability: 2.500±0.66 -> 2.800±0.43 | Helped |
| Add expert, task, and tone preconditioning to a student-question-only prompt (ablation) | zero-shot · DATA 100 stand-alone questions | Tone: 2.390±0.62 -> 2.820±0.41 | Helped |
| Add expert, task, and tone preconditioning to a student-question-only prompt (ablation) | zero-shot · DATA 100 stand-alone questions | Factuality: 1.920±0.84 -> 1.860±0.80 | Helped |
| Add Q&A and course-document retrieval context plus engineered prompting (ablation) | zero-shot · DATA 100 stand-alone questions | ROUGE 1: 0.136 -> 0.193 | Helped |
Surprising findings
- Zero-shot LLM responses often hallucinated, such as fabricating questions or answering as if responding to emails.
- The no-retrieval baseline sometimes hallucinated course policies and resources not used in the course.
Limitations
- Evaluated only in a single course (DATA 100), so generalization is unproven.
- Only stand-alone questions were evaluated; mega-thread questions were not.
- When retrieval fails, the system responds using incorrect documents or Q&A pairs.
Experimental setup
Datasets: DATA 100 EdSTEM Q&A pairs (Fall 2022 and Spring 2023, 6016 pairs) DATA 100 course documents (notes, textbook, syllabus, policies, worksheets; homework solutions excluded)
Benchmarks: DATA 100 EdSTEM test set (100 stand-alone questions)
Metrics: ROUGE-1 ROUGE-2 BERTScore human relevance readability positive tone factuality
Baselines: zero-shot LLaMA-2-13B-chat no-retrieval prompting no-LLM retrieval only
Builds on: Retrieval-augmented generation Chain-of-thought prompting Instructor embeddings multi-qa-mpnet-base-dot-v1
Closest papers by meaning
- Putting People in LLMs’ Shoes: Generating Better Answers via Question Rewriter (AAAI 2025 · similarity 0.88)
- SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs (ICLR 2024 · similarity 0.86)
- LLMs in the Classroom: Outcomes and Perceptions of Questions Written with the Aid of AI (AAAI 2025 · similarity 0.86)
- MerryQuery: A Trustworthy LLM-Powered Tool Providing Personalized Support for Educators and Students (AAAI 2025 · similarity 0.86)
- RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content (NeurIPS 2024 · similarity 0.86)
- Automated Evaluation of Retrieval-Augmented Language Models with Task-Specific Exam Generation (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.