Research map / Large language models / Reasoning and chain of thought

Amortizing intractable inference in large language models

ICLR 2024 · Large language models › Reasoning and chain of thought · approach: Theory of autoregressive reasoning · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes fine-tuning LLMs with GFlowNet (subtrajectory balance) objectives to amortize sampling from intractable posteriors over continuations, latent chain-of-thought rationales, and tool-use sequences.

Problem

Autoregressive LLMs only support tractable left-to-right sampling, while infilling, constrained generation, sequence continuation, and latent chain-of-thought reasoning require sampling from intractable posterior distributions; reward-maximizing RL and MCMC either collapse to modes or are slow.

Approach

Initialize a GFlowNet policy from a pretrained LLM and train it with a modified subtrajectory balance objective so that sampled sequences are proportional to an LM-computed reward (e.g., p_LM(XZY)), amortizing posterior inference over inputs; combined with EM-style SFT this gives a latent-variable view of chain-of-thought.

Key results

• On SUBJ with GPT-J 6B and 10 labeled examples, GFlowNet fine-tuning reaches 71.4% vs 64.3% for supervised fine-tuning (75.2% with SFT on top).
• On integer arithmetic with 3 operands, GFlowNet fine-tuning achieves 95.2% accuracy vs 72.1% for SFT and 30.6% for PPO; OOD 5-operand accuracy is 40.7% vs 12.8% for SFT.
• On story infilling, GFlowNet fine-tuning gets BERTScore 0.184 and GPT-4 coherence 3.4, versus 0.094 and 2.7 for supervised fine-tuning.

Measured effects reported in the paper

Surprising findings

• PPO-trained policies on the arithmetic task produce high-reward sequences that do not even form valid tool calls, indicating reward overoptimization.
• On the random-number task, PPO raises valid-number rate to 95.8% but leaves the distribution highly skewed, while GFlowNet fine-tuning reaches KL 9.75e-5 from uniform.

Limitations

• Experiments limited to models up to 6B parameters.
• Exploration remains an open problem for on-policy training with more complex latents.
• Improves inference over existing knowledge but does not address hallucination or miscalibration.

Experimental setup

Datasets: OpenWebText ROCStories SUBJ synthetic arithmetic expressions

Benchmarks: SUBJ ROCStories infilling integer arithmetic (3-5 operands) OpenWebText continuation prompts

Metrics: test accuracy BERTScore BLEU-4 GLEU-4 GPT-4 coherence rating log-likelihood embedding cosine diversity

Baselines: beam search diverse beam search nucleus sampling few-shot prompting supervised fine-tuning PPO zero-shot chain-of-thought prompting

Builds on: GFlowNet subtrajectory balance PPO LoRA expectation-maximization

Closest papers by meaning

• Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening (ICML 2026 · similarity 0.90)
• Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models (ICLR 2025 · similarity 0.89)
• BRAIn: Bayesian Reward-conditioned Amortized Inference for natural language generation from feedback (ICML 2024 · similarity 0.89)
• PowerFlow: Unlocking the Dual Nature of LLMs via Principled Distribution Matching (ICML 2026 · similarity 0.89)
• Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models (ICLR 2026 · similarity 0.88)
• Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages (NeurIPS 2026 · similarity 0.88)