Research map / Large language models / Instruction and fine-tuning

Quantifying Prediction Consistency Under Fine-tuning Multiplicity in Tabular LLMs

ICML 2025 · Large language models › Instruction and fine-tuning · approach: Fine-tuning generalization and delta compression · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a perturbation-based "local stability" score, computed from a single fine-tuned tabular LLM by sampling neighbors in embedding space, to preemptively flag predictions likely to be inconsistent under fine-tuning multiplicity, with a probabilistic consistency guarantee.

Problem

Fine-tuned LLMs on tabular classification can yield equally accurate models that disagree on the same inputs (fine-tuning multiplicity); measuring this by retraining many models is computationally prohibitive.

Approach

Defines local stability as the mean confidence of a model over k points sampled on a hypersphere around an input in embedding space, minus the mean absolute deviation of confidence from the input's confidence, and proves a Hoeffding-based probabilistic bound linking this score to predictions under a class of equally well-performing fine-tuned models.

Key results

• On the Adult 128-shot T0 setting, local stability reaches Spearman 0.96 with pairwise disagreement versus 0.62 for prediction probability and 0.83 for drop-out.
• On Hospital Readmission, stability separates correct (0.8710) from incorrect (0.5729) predictions more than prediction probability (0.8994 vs 0.7965) does.
• Stability correlates better than the drop-out baseline in most dataset/metric combinations, but not uniformly (e.g., German Credit arbitrariness 0.54 vs 0.57 for prediction probability).

Measured effects reported in the paper

Surprising findings

• Prediction confidence alone does not guarantee consistency: confident predictions can still be unstable, with 20% of Hospital Readmission test predictions confident but unstable.
• Fine-tuned LLM decision boundaries show abrupt, noisy variations even in regions of high confidence on synthetic 2D data.

Limitations

• The probabilistic bound depends on an epsilon' term that is not directly computable, so the guarantee is not practically checkable.
• Requires access to the embedding space, limiting use to open-weight models.
• Performance is sensitive to sample size k and neighborhood radius sigma; applicability to high-dimensional tables is constrained by context window.

Experimental setup

Datasets: Adult German Credit Diabetes Bank Heart Car Hospital Readmission

Benchmarks: Adult German Credit Diabetes Bank Heart Car Hospital Readmission

Metrics: Arbitrariness Discrepancy Pairwise Disagreement Prediction Variance Prediction Range Spearman correlation

Baselines: Prediction probability Binary Drop-Out Method Adversarial Weight Perturbation (AWP) Re-training

Builds on: LoRA T-Few Dropout-based Rashomon set exploration Adversarial Weight Perturbation

Closest papers by meaning

• Strengthening LLMs for Tabular Prediction with Structural Priors (NeurIPS 2026 · similarity 0.88)
• Retrieval & Fine-Tuning for In-Context Tabular Models (NeurIPS 2024 · similarity 0.87)
• TabArena: A Living Benchmark for Machine Learning on Tabular Data (NeurIPS 2025 · similarity 0.85)
• Mitigating Forgetting in LLM Fine-Tuning via Low-Perplexity Token Learning (NeurIPS 2025 · similarity 0.85)
• One Algorithm, Two Goals: Dual Scoring for Parameter and Data Selection in LLM Fine-Tuning (NeurIPS 2026 · similarity 0.85)
• Quantifying Uncertainty in Error Consistency: Towards Reliable Behavioral Comparison of Classifiers (NeurIPS 2025 · similarity 0.85)