Research map / Large language models / Reasoning and chain of thought

Latent Self-Consistency for Reliable Majority-Set Selection in Short- and Long-Answer Reasoning

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Latent Self-Consistency (LSC), which trains a few learnable summary tokens appended after each sampled response (frozen base LLM, supervised contrastive loss) to select the majority-consistent response across short- and long-form tasks with negligible overhead.

Problem

Self-consistency relies on exact string matching and fails for long-form answers, while existing extensions (USC, WUCS) are costly or lose accuracy on short-answer tasks.

Approach

Append trainable summary tokens after each candidate's EOS, reuse the KV cache to get response embeddings, train them only (base frozen) with supervised contrastive loss on pseudo-labeled answers, and select the response with highest exponentially weighted cosine similarity to others, optionally with a dynamic Top-K neighborhood size.

Key results

• On short-answer tasks LLaMA3.1-8B-Instruct, LSC+Dynamic TopK reaches 72.8% average vs 72.2% for SC and 67.7% for USC.
• LSC adds at most 0.9% runtime overhead versus USC's roughly 7-18% time overhead across models.
• LSC achieves 97.8% consistency on short-answer tasks and 92.4% on long-answer tasks, versus about 88% for WUCS and USC.

Surprising findings

• Off-the-shelf sentence embeddings (SBERT) are clearly weaker than LSC on short-answer tasks, though competitive on long-answer tasks.
• Some settings show USC or WUCS achieving slightly higher accuracy than LSC despite much lower consistency, which the authors attribute to scoring-rule inductive biases.

Limitations

• Training data is small and relies on LLM-generated pseudo-labels from the same base models.
• Long-answer consistency labels and TruthfulQA truth scores depend on GPT-4.1 as judge.
• Gains on some benchmarks are small and within a fraction of a point of baselines; dynamic Top-K does not uniformly help.

Experimental setup

Datasets: GSM8K MATH TriviaQA MMLU CommonsenseQA TruthfulQA MSMARCO-NLG HumanEval MBPP CNN/DailyMail

Benchmarks: GSM8K MATH TriviaQA MMLU CommonsenseQA TruthfulQA MSMARCO-NLG HumanEval MBPP CNN/DailyMail

Metrics: accuracy BLEU ROUGE BERTScore Pass@1 consistency expected calibration error time overhead memory overhead

Baselines: Self-Consistency Universal Self-Consistency WUCS Fine-grained Self-Consistency SBERT Fine-tuned SBERT EoS token variants

Builds on: Self-Consistency Universal Self-Consistency Weighted Unigram Consistency Score supervised contrastive learning prompt tuning

Closest papers by meaning

• Optimal Self-Consistency for Efficient Reasoning with Large Language Models (ICML 2026 · similarity 0.91)
• Representation Consistency for Accurate and Coherent LLM Answer Aggregation (NeurIPS 2025 · similarity 0.91)
• Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs (ICLR 2026 · similarity 0.90)
• CITE: Anytime Valid Statistical Inference in LLM Self-Consistency (NeurIPS 2026 · similarity 0.90)
• A Theoretical Study on Bridging Internal Probability and Self-Consistency for LLM Reasoning (NeurIPS 2025 · similarity 0.90)
• Scalable Best-of-N Selection for Large Language Models via Self-Certainty (NeurIPS 2025 · similarity 0.89)