Research map / Large language models / Language, safety and interpretability
The Generalization and Robustness of Transformer-Based Language Models on Commonsense Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
A doctoral-thesis-style summary of three studies on RoBERTa and ChatGPT for commonsense QA: cross-benchmark generalization, confidence behavior under ambiguous/perturbed prompts, and a risk-adjusted calibrator (DwD) with risk metrics.
Problem
Commonsense reasoning benchmark scores may overestimate LLM ability because models may not generalize across benchmarks and their confidence may not reflect risk under ambiguous inference tasks.
Approach
Measures cross-benchmark generalization via a performance loss metric with significance testing, probes confidence with prompt- and choice-based perturbations, and proposes a risk-adjusted calibrator (DwD) that separates decision and composite risks.
Key results
- DwD reduces in-domain and out-of-domain decision risks for RoBERTa by 28.3% and 18.9%, respectively.
- DwD reduces additional composite risk of ChatGPT NLI by 17.9% compared to the next best baseline.
- RoBERTa fine-tuned on one commonsense benchmark shows significant performance decrease when tested on another, indicating poor generalization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply DwD risk-adjusted calibration | RoBERTa · In-domain decision risk | decision risk: 28.3% | Helped |
| Apply DwD risk-adjusted calibration | RoBERTa · Out-of-domain decision risk | decision risk: 18.9% | Helped |
| Apply DwD risk-adjusted calibration | ChatGPT NLI · NLI | additional composite risk: 17.9% | Helped |
| Fine-tune on one commonsense benchmark and test on a different benchmark | RoBERTa · Five commonsense reasoning benchmarks | performance | Hurt |
| Apply prompt-based confusion perturbations | Fine-tuned language model · Commonsense multiple-choice QA benchmarks | confidence distribution of originally incorrect answers | No clear effect |
| Apply choice-based perturbation | Fine-tuned language model · Ambiguous inference task | answer correctness | Hurt |
Surprising findings
- When prompts are perturbed, the confidence distribution of originally incorrect answers stays close to random, and the model still prefers the originally correct (now pseudo-correct) answer.
Limitations
- Generalization findings not verified for GPT-series generative models.
- Reliance on multiple-choice format may not reflect real-world conversational interaction.
- Evaluation limited to human-free automated metrics; human-in-the-loop evaluation not done.
Experimental setup
Datasets: five commonsense reasoning benchmarks (unnamed in text)
Benchmarks: five commonsense multiple-choice QA benchmarks
Metrics: performance loss decision risk composite risk risk sensitivity
Baselines: next best calibration baseline
Closest papers by meaning
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.88)
- Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Cultures (NeurIPS 2026 · similarity 0.88)
- MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation (ICLR 2025 · similarity 0.88)
- MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models (AAAI 2026 · similarity 0.87)
- Linguini: A benchmark for language-agnostic linguistic reasoning (NeurIPS 2025 · similarity 0.87)
- Improving Factuality and Reasoning in Language Models through Multiagent Debate (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.