Research map / Large language models / Instruction and fine-tuning
LitCab: Lightweight Language Model Calibration over Short- and Long-form Responses
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LitCab, a lightweight calibration layer (a single linear layer predicting logit biases trained with a max-margin objective) and the LitCab benchmark spanning phrase-, sentence-, and paragraph-level generation tasks for evaluating LM calibration.
Problem
Language model confidence is often miscalibrated, and standard calibration methods either cannot reorder candidate generations (temperature scaling) or require full-model fine-tuning, which is impractical at LM scale; calibration over long-form outputs is also under-studied.
Approach
Train a single linear layer on the LM's last-layer hidden states to predict a bias added to the output logits, using a contrastive max-margin loss over correct vs. incorrect generations, so confidence rankings change while the LM stays frozen; paragraph-level calibration is evaluated per claim.
Key results
- LitCab lowers average ECE to 0.093 from 0.137 for the original Llama2-7B across the eight tasks, with NQ ECE dropping from 0.171 to 0.101.
- LitCab adds fewer than 2% of the original model's parameters and outperforms label smoothing (avg ECE 0.191) and temperature scaling (avg ECE 0.145) on average.
- Larger models in the same family are better calibrated on phrase-level tasks, but LLaMA-30B is not better calibrated on sentence- and paragraph-level tasks; GPT2-XL (1.5B) is better calibrated than larger models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply LitCab calibration | Llama2-7B · NQ | ECE: 0.171 -> 0.101 | Helped |
| apply LitCab calibration | Llama2-7B · Average across suite | ECE: 0.137 -> 0.093 | Helped |
| apply LitCab calibration | Llama2-7B · Average across suite | Brier: 0.213 -> 0.197 | Helped |
| add temperature scaling to LitCab (ablation) | Llama2-7B with LitCab · phrase- and sentence-level tasks | ECE | Helped |
| add temperature scaling to LitCab (ablation) | Llama2-7B with LitCab · paragraph-level tasks | ECE | Hurt |
| increase model size within LLaMA or Llama2 family | open-source LMs · phrase-level tasks | ECE | Helped |
| increase model size within LLaMA or Llama2 family | open-source LMs · sentence- and paragraph-level tasks | ECE | Mixed |
| fine-tune LLaMA-13B on user-shared conversations | LLaMA-13B · CaT | ECE | Hurt |
| fine-tune Llama2-7B with label smoothing | Llama2-7B · Average across suite | ECE: 0.137 -> 0.191 | Hurt |
| use verbalized confidence instead of the original model confidence | Llama2-7B · most tasks | ECE | Hurt |
Surprising findings
- GPT2-XL (1.5B) is better calibrated than larger models such as LLaMA-30B despite lower accuracy.
- Vicuna-13B, fine-tuned from LLaMA-13B, is worse calibrated than its base model.
Limitations
- Paragraph-level tasks rely on GPT-3.5-turbo for claim extraction and span mapping and GPT-4 for correctness judgment.
- Verbalization and self-consistency cannot be evaluated at claim level, so paragraph-level comparisons are incomplete.
- Label smoothing baseline with LoRA may be underperforming due to small training data and limited tuning.
Experimental setup
Datasets: NaturalQuestions SciQ TriviaQA TruthfulQA WikiQA BioGen WikiGen QAMPARI FEVER WikiBio
Benchmarks: NaturalQuestions SciQ TriviaQA TruthfulQA WikiQA BioGen WikiGen QAMPARI
Metrics: ECE Brier score acc@q cov@p
Baselines: Temperature Scaling Label Smoothing P(IK) Verbalization Self-Consistency
Builds on: temperature scaling label smoothing P(IK) self-consistency
Closest papers by meaning
- Thermometer: Towards Universal Calibration for Large Language Models (ICML 2024 · similarity 0.90)
- Calibrating Large Language Models with Sample Consistency (AAAI 2025 · similarity 0.90)
- Beware of Calibration Data for Pruning Large Language Models (ICLR 2025 · similarity 0.89)
- Calibrated Language Models and How to Find Them with Label Smoothing (ICML 2025 · similarity 0.88)
- Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach (ICML 2025 · similarity 0.88)
- Paloma: A Benchmark for Evaluating Language Model Fit (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.