Research map / Large language models / Reasoning and chain of thought

GRACE: Generative Representation Learning via Contrastive Policy Optimization

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Trains instruction-tuned LLMs as policies that generate rationales, pools hidden states of the rationale into embeddings, and optimizes contrastive-derived rewards with GRPO-style policy gradients, evaluated on MTEB.

Problem

Contrastive training of LLM text encoders treats the model as a black-box encoder, discarding its generative and reasoning abilities and yielding opaque static embeddings.

Approach

Treat contrastive similarity signals as rewards over model-generated rationales rather than as losses; the LLM generates rationales for queries and documents, the rationale hidden states are mean-pooled into embeddings, and policy gradient (GRPO) updates the generator with a composite reward (contrastive margin, within-document consistency, and batch hard-negative terms).

Key results

• Supervised GRACE on Qwen2.5-1.5B-Instruct averages 45.48 MTEB vs 30.33 base; averaged over four backbones the supervised setting improves the MTEB average by 11.52% over base.
• Unsupervised GRACE averages 6.85% above base across the four backbones, with Qwen3-4B unsupervised reaching 51.03 vs 45.49 base.
• Supervised CL fine-tuning drops general-domain average on GSM8K/MMLU/HumanEval etc. to about 8% (near zero on GSM8K, MMLU, BBH, HumanEval), while GRACE stays within about ±0.4 of base.

Surprising findings

• Naive contrastive (CL) fine-tuning collapses general-domain performance to near zero on GSM8K, MMLU, BBH and HumanEval, whereas the RL-based method preserves them.
• Explicit reasoning alone (Base w/ reasoning, no training) yields a modest gain over base embeddings.

Limitations

• Generation-based embeddings are much slower than encoder-only models because generation dominates latency.
• Theoretical analysis (InfoNCE connection, convergence) is loosely argued and not rigorous.
• Reported gains are on a text-embedding benchmark with a single training dataset replication and four backbones.

Experimental setup

Datasets: E5 dataset (replication, 1.5M samples) SimCSE unsupervised setup data

Benchmarks: MTEB (56 datasets; 16-task subset for ablations) GSM8K MMLU TriviaQA FEVER BBH HumanEval

Metrics: nDCG@10 MAP V-measure accuracy average precision Spearman correlation EM Pass@1

Baselines: Base (no training) Base w/ reasoning Contrastive learning (InfoNCE) BERT RoBERTa LLM2Vec Echo SimCSE ReMax REINFORCE++ DAPO

Builds on: GRPO InfoNCE SimCSE DAPO REINFORCE++

Closest papers by meaning

• Generative Representational Instruction Tuning (ICLR 2025 · similarity 0.87)
• UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddings (ICLR 2026 · similarity 0.87)
• Scaling Language-centric Omnimodal Representation Learning (NeurIPS 2025 · similarity 0.87)
• Aligning Visual Contrastive learning models via Preference Optimization (ICLR 2025 · similarity 0.86)
• Learning Goal-Conditioned Representations for Language Reward Models (NeurIPS 2024 · similarity 0.85)
• Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe (NeurIPS 2024 · similarity 0.85)