Research map / Label-efficient and robust learning / Recognition and long-tail

Using Next Sentence Prediction to Test ChatGPT’s Text Comprehension (Student Abstract)

AAAI 2025 · Label-efficient and robust learning › Recognition and long-tail · approach: Multilingual language representation pretraining

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Next Sentence Prediction (NSP) as an automated, scalable test of ChatGPT's text comprehension on English and Swahili children's stories, and analyzes which text features predict its errors.

Problem

Testing whether LLMs comprehend text in a simple, objective, scalable, automatically scored way, especially across languages, for educational tutoring use (RoboTutor).

Approach

Frame inter-sentential comprehension as distinguishing the true next story sentence from a distractor drawn from later in the same story, so labels are automatic by definition; then regress ChatGPT's correctness on controlled and observed text features.

Key results

• ChatGPT (gpt-4-turbo) answered 88% of 12,674 English NSP questions correctly, above 50% chance.
• ChatGPT answered 79% of 1,010 Swahili NSP questions correctly, not significantly above chance (p=0.96).
• Logistic regression predicted ChatGPT's correctness on held-out English stories with 94% accuracy; distractor length (logit -0.057) and distractor distance (logit 0.063) were significant predictors.

Measured effects reported in the paper

Surprising findings

• Shorter distractors were easier for ChatGPT even though they carry less information.
• Swahili accuracy of 100% on the 101-question held-out test set was not statistically significant versus chance due to small sample size.

Limitations

• Swahili data too small for statistical reliability.
• Results limited to children's narrative fiction and may not generalize to other genres.
• Assumes statistical independence of NSP questions sharing the same story context.

Experimental setup

Datasets: African Storybook Project English stories (218) African Storybook Project Swahili stories (15 sampled)

Benchmarks: NSP questions from African Storybook English stories NSP questions from African Storybook Swahili stories

Metrics: percentage correct predictive accuracy logit coefficients p-value from t-test

Baselines: decision trees gradient boosting linear regression random forest XGBoost

Builds on: Feng and Mostow 2021 NSP question generation

Closest papers by meaning

• Prerequisite Skills for Reading Comprehension: Multi-Perspective Analysis of MCTest Datasets and Systems (AAAI 2017 · similarity 0.84)
• Words or Characters? Fine-grained Gating for Reading Comprehension (ICLR 2017 · similarity 0.83)
• Text Alignment Is An Efficient Unified Model for Massive NLP Tasks (NeurIPS 2023 · similarity 0.83)
• LatestEval: Addressing Data Contamination in Language Model Evaluation through Dynamic and Time-Sensitive Test Construction (AAAI 2024 · similarity 0.82)
• ChatQA: Surpassing GPT-4 on Conversational QA and RAG (NeurIPS 2024 · similarity 0.81)
• SeqGPT: An Out-of-the-Box Large Language Model for Open Domain Sequence Understanding (AAAI 2024 · similarity 0.81)