Research map / Large language models / Reasoning and chain of thought

Strengthening LLMs for Tabular Prediction with Structural Priors

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Routing, pruning and distillation for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PRPO, a GRPO-style RL post-training method for LLMs that uses label-preserving column permutations and two-level (intra/inter-permutation) advantages to densify sparse rewards for tabular prediction, yielding an 8B Qwen3 model competitive with tabular baselines.

Problem

LLMs struggle to be competitive on tabular prediction because outcome-level RL rewards are sparse and LLMs fail to capture tabular structure such as column-permutation invariance.

Approach

Exploit column-permutation invariance of tabular data: sample several label-preserving column permutations per example, roll out G candidates per permutation, and combine intra-permutation (local) and global (pooled) group-relative advantages to densify rewards within a GRPO-style objective.

Key results

• Fully supervised classification mean accuracy 0.8436 for PRPO-Qwen3-8B vs 0.8413 for TabPFN and 0.8234 for XGBoost.
• Cross-dataset zero-shot transfer accuracy 0.7021 on 53 unseen classification datasets, above 32-shot TabPFN (0.6974); with 32 shots it reaches 0.7542.
• Ablation: full PRPO improves over the random column-order variant by +0.0780 classification accuracy; fully supervised regression is weaker than TabPFN (0.1499 vs 0.1236 NMAE).

Surprising findings

• Fully supervised PRPO-Qwen3-8B trails TabPFN on regression NMAE (0.1499 vs 0.1236).
• Raw zero-shot DeepSeek-R1 (685B) reaches only about 57% on binary classification, and general LLMs underperform tabular models even with 32 shots.

Limitations

• Regression performance remains behind TabPFN and CatBoost in the fully supervised setting.
• Theoretical claims rely on idealized assumptions (e.g., independence of rollouts, a well-trained invariant policy), and the variance proof has questionable steps.
• Zero-shot results on regression are much weaker than 32-shot, and only 8B-scale base models with 5 rollouts were explored.

Experimental setup

Datasets: 139 OpenML datasets (103 classification, 36 regression)

Benchmarks: 139 OpenML datasets (fully supervised: 50 classification, 15 regression; zero-shot/transfer and 32-shot: 53 classificati

Metrics: accuracy NMAE win/tie/loss

Baselines: XGBoost LightGBM CatBoost TabPFN TabLLM Tabula-8B Qwen2.5-7B Qwen2.5-32B Qwen3-8B Qwen3-32B Qwen3-235B DeepSeek-R1

Builds on: GRPO DeepSeek-R1 Verl TabLLM TabPFN

Closest papers by meaning

• Large Scale Transfer Learning for Tabular Data via Language Modeling (NeurIPS 2024 · similarity 0.89)
• TabDPT: Scaling Tabular Foundation Models on Real Data (NeurIPS 2025 · similarity 0.88)
• Quantifying Prediction Consistency Under Fine-tuning Multiplicity in Tabular LLMs (ICML 2025 · similarity 0.88)
• Understanding the Surprising Generalization Properties of Tabular Foundation Models (NeurIPS 2026 · similarity 0.88)
• Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning (ICML 2024 · similarity 0.88)
• Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning (NeurIPS 2024 · similarity 0.87)