Research map / Vision-language and multimodal / CLIP and zero-shot
Meaning Representations from Trajectories in Autoregressive Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Represents a text's meaning as the distribution over its sampled continuations (trajectories) from an autoregressive LM, enabling prompt-free, fine-tuning-free similarity and asymmetric entailment/hyponym tests.
Problem
Autoregressive LLMs do not produce explicit sentence representations, and existing extraction methods rely on prompts, fine-tuning, or vector embeddings that cannot model asymmetric (entailment/hypernym) relations.
Approach
Define a string's meaning as the likelihood function over all its possible continuations; estimate semantic distance by Monte Carlo sampling of trajectories from both strings and comparing their log-likelihoods, and use min (meet) over these functions to test containment and entailment direction.
Key results
- Ours (LLaMA-33B) achieves 66.6 average Spearman on STS tasks, versus 58.0 for the best zero-shot encoder (ST5-Enc-mean 11B) and 65.7 for Ours (Falcon-7B), without fine-tuning or prompts.
- On CxC image-image similarity, Ours with LLaVA reaches 81.47 vs 66.33 for CLIP-ViTL/14; with an alignment prompt, CxC-SITS rises to 67.63 vs 64.25 for CLIP-ViTL/14.
- Entailment Test on SNLI-Entailment outperforms random and likelihood baselines by 15.7% and 9.3% respectively using Falcon-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Represent sentence meaning with trajectory distributions instead of Falcon-7B cross-encoder scoring | Falcon-7B · STS benchmarks (Avg) | Spearman correlation ×100: 47.5 -> 65.7 | Helped |
| Represent image meaning with trajectory distributions instead of CLIP-ViTB/16 vision embeddings | LLaVA · CxC-SIS | Spearman correlation ×100: 72.08 -> 81.47 | Helped |
| Add alignment prompts for image and caption inputs (ablation) | LLaVA trajectory method · CxC-SITS | Spearman correlation ×100: 57.14 -> 67.63 | Helped |
| Replace Log-L1 with Hellinger distance (ablation) | LLaMA-13B trajectory method · STS-B validation | Spearman correlation ×100: 70.6 -> 69.7 | Hurt |
| Replace Log-L1 with total variation distance (ablation) | LLaMA-13B trajectory method · STS-B validation | Spearman correlation ×100: 70.6 -> 51.2 | Hurt |
| Replace Log-L1 with symmetric KL divergence (ablation) | LLaMA-13B trajectory method · STS-B validation | Spearman correlation ×100: 70.6 -> 69.7 | Hurt |
| Replace Log-L1 with Log-L2 (ablation) | LLaMA-13B trajectory method · STS-B validation | Spearman correlation ×100: 70.6 -> 69.2 | Hurt |
| Append a full stop to incomplete sentences (ablation) | Falcon-7B trajectory method · STS-B | Spearman correlation ×100: 67.2 -> 67.7 | Helped |
| Use a fixed trajectory set with n=20 instead of pair-specific trajectories (ablation) | Falcon-7B trajectory method · STS-B validation | Spearman correlation ×100: 74.74 -> 49.41 | Hurt |
| Use a fixed trajectory set with n=40 instead of pair-specific trajectories (ablation) | Falcon-7B trajectory method · STS-B validation | Spearman correlation ×100: 74.74 -> 53.29 | Hurt |
| Increase the number and length of sampled trajectories (ablation) | Falcon-7B trajectory method · STS-B validation | Spearman correlation | Helped |
| Use sampling temperature values that are too high or too low instead of λ=1.0 (ablation) | Falcon-7B trajectory method · STS-B validation | Spearman correlation | Hurt |
Surprising findings
- Prompt-free likelihood-based baselines (last/mean token) perform far below the proposed method, and the cross-encoder conditional likelihood is weaker than expected on image-text similarity.
- Using a fixed set of trajectories for all comparisons (for semantic search) drops STS-B validation Spearman from 74.74 to 49.41 with n=20.
Limitations
- Computationally more expensive than single-forward-pass embedding methods; semantic search needs per-query trajectory sampling.
- Still well below contrastively fine-tuned models such as SimCSE on STS.
- Hyponym and entailment tests use approximations (meet computed on the union of sampled trajectories) and performance on WordNet is only partially above chance.
Experimental setup
Datasets: STS-B STS12 STS13 STS14 STS15 STS16 SICK-R SNLI WordNet WikiText Crisscrossed Captions (CxC)
Benchmarks: STS-B STS12 STS13 STS14 STS15 STS16 SICK-R SNLI-Entailment WordNet hyponym/hypernym CxC-SIS CxC-SITS
Metrics: Spearman correlation (x100) accuracy
Baselines: Cross Encoder (conditional likelihood) Joint Likelihood Last token Mean token CLIP SimCSE ST5-Enc-mean (11B) PromptEOL PromptBERT PromptRoBERTa
Builds on: distributional hypothesis PromptEOL Sentence-BERT word2vec
Closest papers by meaning
- Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flows (NeurIPS 2025 · similarity 0.86)
- Invariant Features in Language Models: Geometric Characterization and Model Attribution (NeurIPS 2026 · similarity 0.84)
- Causal Graphical Models for Vision-Language Compositional Understanding (ICLR 2025 · similarity 0.84)
- Lines of Thought in Large Language Models (ICLR 2025 · similarity 0.84)
- Explaining Datasets in Words: Statistical Models with Natural Language Parameters (NeurIPS 2024 · similarity 0.84)
- On the Origins of Linear Representations in Large Language Models (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.