Research map / Label-efficient and robust learning / Recognition and long-tail
The Truth Lies Somewhere in the Middle (of the Generated Tokens)
Sophie L. Wang, Phillip Isola, Brian Cheung
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| mean-pool hidden states over generated tokens instead of using any single token (ablation) | Qwen3-14B generated-token embeddings · WIT vision-language alignment | CKA alignment | Improved |
| mean pooling versus attention or max pooling (ablation) | generated-token embeddings · WIT vision-language alignment | CKA alignment | Improved |
| mix representations from different token segments via convex combinations (ablation) | generated-token embeddings · WIT vision-language alignment | CKA alignment | Improved |
| mix generated-token representations on reasoning and protein tasks (ablation) | Qwen3-14B · MATH-500, GPQA Diamond, UniProt | CKA alignment | Improved |
| use generated tokens instead of prompt tokens (ablation) | Qwen3-14B mean-pooled embeddings · WIT vision-language alignment | CKA alignment: 0.184 -> 0.410 | Improved |
| mean pool prompt tokens instead of using later individual prompt tokens (ablation) | prompt-token embeddings · WIT vision-language alignment | CKA alignment | No clear effect |
| mix representations across independent decoding seeds (ablation) | generated-token embeddings · WIT vision-language alignment | CKA alignment | Improved |
| mix region-conditioned generations describing distinct parts of an image (ablation) | generated-token embeddings · DCI vision-language alignment | CKA alignment | Improved |
| add isotropic noise around the mean embedding / shuffle image-text pairs as variance-reduction control (ablation) | token-mean representation · WIT vision-language alignment | CKA alignment | No clear effect |
| prepend the phrase 'Let me recall what I know' to a non-thinking generation (ablation) | Qwen3-14B generation · WIT vision-language alignment | tokenwise alignment | Improved |
| embed text generated by a different model (ablation) | Qwen3-14B and OLMo3-7B-Think · WIT vision-language alignment | tokenwise alignment | Hurt |
| embed Qwen3-generated text with OLMo3 instead of OLMo3's own generations (ablation) | OLMo3-7B-Think embeddings · WIT vision-language alignment | CKA alignment | Improved |
Closest papers by meaning
- From Bytes to Ideas: Language Modeling with Autoregressive U-Nets (NeurIPS 2025 · similarity 0.85)
- Effective Context in Transformers: An Analysis of Fragmentation and Tokenization (NeurIPS 2026 · similarity 0.85)
- Transformers Represent Belief State Geometry in their Residual Stream (NeurIPS 2024 · similarity 0.85)
- Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Space (NeurIPS 2024 · similarity 0.85)
- Blink of an eye: a simple theory for feature localization in generative models (ICML 2025 · similarity 0.84)
- Tracing the Representation Geometry of Language Models from Pretraining to Post-training (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
