Research map / Large language models / Language, safety and interpretability

Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineering

ICLR 2024 · Large language models › Language, safety and interpretability · approach: In-context learning analysis · 8 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Batch Calibration (BC), a zero-shot inference-only method that removes contextual bias by subtracting the batch-marginal LLM output, plus a single-parameter labeled extension (BCL), evaluated on PaLM 2 and CLIP.

Problem

LLM prompting and in-context learning are sensitive to contextual bias from templates, verbalizers, and demonstrations; existing calibration methods (CC, DC, PC) have failure cases, especially on multi-sentence tasks.

Approach

Estimate the contextual bias term as the mean of LLM output scores over the unlabeled test batch and subtract it from each prediction (linear decision boundary shift), with an optional learned strength gamma for a few-shot variant (BCL).

Key results

• On 1-shot PaLM 2-S, BC averages 74.41% accuracy vs 66.20% for ICL and 67.65% for PC across 13 NLU tasks.
• On 1-shot PaLM 2-L, BC averages 81.09% vs 75.16% ICL and 77.83% CC.
• Zero-shot CLIP ViT-B/16 with BC improves average accuracy on image tasks from 36.7% to 41.5%.

Measured effects reported in the paper

Surprising findings

• Content-free-token calibration (CC) and random-text calibration (DC) can introduce additional bias on multi-sentence tasks such as QNLI, where the N/A token is read as entailed content.
• PC's EM-GMM boundary is unstable, with large standard deviations (e.g., 17.73 on QNLI for PaLM 2-S) and frequent substantial degradation.

Limitations

• BC requires a batch of unlabeled test inputs to estimate the contextual bias, so it is a test-time batch method.
• Evaluated only on classification tasks; extension to generative tasks is left to future work.

Experimental setup

Datasets: SST-2 MNLI QNLI MRPC QQP BoolQ CB COPA RTE WiC ANLI-R1 ANLI-R2

Benchmarks: SST-2 MNLI QNLI MRPC QQP BoolQ CB COPA RTE WiC ANLI-R1 ANLI-R2

Metrics: accuracy

Baselines: ICL Contextual Calibration Domain-Context Calibration Prototypical Calibration zero-shot CLIP

Builds on: contextual calibration domain-context calibration prototypical calibration

Closest papers by meaning

• C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion (ICLR 2024 · similarity 0.89)
• Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting (ICLR 2024 · similarity 0.88)
• Open-Vocabulary Calibration for Fine-tuned CLIP (ICML 2024 · similarity 0.88)
• Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models (ICML 2025 · similarity 0.88)
• Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach (ICML 2025 · similarity 0.87)
• Thermometer: Towards Universal Calibration for Large Language Models (ICML 2024 · similarity 0.87)