Research map / Large language models / Language, safety and interpretability

Enhancing Tabular Learners with Context-Aware Semantic Embeddings

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Controlled text generation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CASE, which continued-pretrains Gemma 3 as a table-language model with target-masked imputation loss, then KV-cache-primes it with sampled rows to extract contextualized row embeddings that are PCA-reduced and appended to features for tabular learners.

Problem

Tabular learners ignore the semantics of textual feature names and cell values, and row-wise or cell-wise LLM embeddings are isolated and not grounded in the dataset's distribution or predictive task.

Approach

Pre-fill the KV cache of a table-tuned Gemma 3 with a random sample of labeled rows so that each target row's final [BOT] hidden state is a context-aware embedding, which is PCA-reduced and concatenated to features for any downstream tabular learner.

Key results

• TabICLv2 with CASE reaches 81.0% accuracy and 77.5 R^2 on CARTE, ahead of AutoGluon (78.8% / 73.7).
• CASE gains range 2.9% to 6.8% accuracy on classification and 3.8% to 19.1% R^2 on regression across CARTE and TextTab for five tabular learners.
• On TabArena (numerics-heavy), TabICLv2 with CASE drops slightly to 88.3% accuracy and 78.5 R^2 versus 88.7% and 79.9 for vanilla TabICLv2.

Surprising findings

• Gains from CASE are statistically significant on semantic benchmarks yet CASE slightly hurts TabArena, with the drop mostly on regression tasks.
• Performance saturates beyond about 1k context tokens, and larger TLM sizes do not benefit from longer context as expected.

Limitations

• Not permutation-equivariant to row or column order, though column-permutation effects were found statistically insignificant.
• Slight performance drop on numerics-heavy TabArena, especially for regression tasks.
• Uses PCA linear projection that may be suboptimal; random context sampling may omit critical anchor rows.

Experimental setup

Datasets: T4 table corpus CARTE TextTab TabArena STRABLE

Benchmarks: CARTE TextTab TabArena STRABLE

Metrics: accuracy R^2 (soft-clipped) rank critical difference win ratio Elo

Baselines: TabICLv2 AutoGluon ConTextTab RealMLP CatBoost TabPFN-2.6 XGBoost TabICL [Cell-Embedding]

Builds on: TabuLa Gemma 3 TabICL ConTextTab TabPFN

Closest papers by meaning

• ConTextTab: A Semantics-Aware Tabular In-Context Learner (NeurIPS 2025 · similarity 0.91)
• Large Scale Transfer Learning for Tabular Data via Language Modeling (NeurIPS 2024 · similarity 0.89)
• TabDPT: Scaling Tabular Foundation Models on Real Data (NeurIPS 2025 · similarity 0.89)
• Understanding the Surprising Generalization Properties of Tabular Foundation Models (NeurIPS 2026 · similarity 0.89)
• TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fields (NeurIPS 2025 · similarity 0.89)
• Benchmarking Attention for Tabular Foundation Models (NeurIPS 2026 · similarity 0.89)