Research map / Large language models / Language, safety and interpretability
Can LLMs Reason Over Non-Text Modalities in a Training-Free Manner? A Case Study with In-Context Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes In-Context Representation Learning (ICRL), a training-free way to feed frozen foundation-model representations (e.g., Uni-Mol molecular embeddings) into a text LLM via PCA strings or untrained, OT-aligned embedding injection, evaluated on molecular property regression.
Problem
Existing methods for giving text LLMs non-text modality representations require costly supervised training of projectors or the LLM; the paper asks whether LLMs can use foundation-model representations in-context without any training.
Approach
Replace text inputs in few-shot ICL examples with representations from an external foundation model, mapped into the LLM via PCA-to-string or an untrained random linear projector whose outputs are distribution-aligned to LLM token embeddings using a closed-form optimal-transport-style mean/variance matching.
Key results
- On ESOL, OT-PCA improves Pearson r by 16.6% over text-only ICL when combined with SMILES input.
- OT-PCA (no text) achieves RMSE 1.140 on ESOL and 1.349 on Lipophilicity, averaging 1.245 across ESOL and Lipo, comparable to some lightweight fine-tuning pipelines (MolecularGPT avg 1.314).
- Activation functions in the untrained projector consistently degrade performance, and standard normal initialization performs best.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ICL to OT-PCA | OT-PCA · ESOL | RMSE: 1.140±0.01 -> 1.094±0.01 | Helped |
| Add ICL to OT-PCA | OT-PCA · Lipo | RMSE: 1.349±0.01 -> 1.277±0.01 | Helped |
| Use text-level PCA strings instead of embedding-level injection | ICRL without original text features · Molecular datasets | RMSE | Helped |
| Use OT alignment rather than Zero-Pad or Random Projection | ICRL without original text features · Molecular datasets | RMSE | Helped |
| Add OT-PCA representations to SMILES input | ICL with SMILES · ESOL | Pearson r: +16.6% | Helped |
| Add PCA representations to SMILES input | ICL with SMILES · Molecular datasets | Performance | Hurt |
| Replace informative FM features with random noise (ablation) | ICRL with original text features · Molecular datasets | Performance | Helped |
| Add Zero-Pad representations to SMILES input | ICL with SMILES · Molecular datasets | Performance | Helped |
| Add activation functions between projector layers (ablation) | ICRL projector · Two evaluated datasets | RMSE | Hurt |
| Use standard normal initialization instead of alternative projector initializations (ablation) | ICRL projector · Two evaluated datasets | RMSE | Helped |
| Increase PCA representation dimensionality (ablation) | Text-level PCA ICRL without text input · Molecular datasets | Performance | Hurt |
| Increase PCA representation dimensionality (ablation) | Text-level PCA ICRL with text input · Molecular datasets | Performance | Helped |
Surprising findings
- Adding SMILES text to ICRL often weakens gains: PCA+ICL degrades performance across all datasets, and OT-based gains are inconsistent with text.
- Zero-Pad and Random Projection often perform worse than random noise baselines.
Limitations
- ICRL underperforms supervised fine-tuned methods that use task-specific training.
- Performance is highly sensitive to LLM capacity; smaller models degrade substantially.
- Evaluated mainly on molecular regression tasks; vision and audio results are only preliminary in the appendix.
Experimental setup
Datasets: ESOL Caco-2 (Caco_wang) AqSolDB LD50_Zhu AstraZeneca lipophilicity TAPE Fluorescence TAPE Stability BindingDB Ki BindingDB IC50
Benchmarks: ESOL Caco_wang AqSolDB LD50_Zhu AstraZeneca TAPE Fluorescence TAPE Stability BindingDB
Metrics: RMSE Pearson r
Baselines: ICL (SMILES text only) Random Noise Zero-Pad Random Projection MolecularGPT GIMLET SELFormer GPT-MolBERTa
Builds on: in-context learning Vector-ICL Uni-Mol Llama-3.1-70B-Instruct
Closest papers by meaning
- Making Text Embedders Few-Shot Learners (ICLR 2025 · similarity 0.89)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.88)
- MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data (NeurIPS 2026 · similarity 0.88)
- Link-Context Learning for Multimodal LLMs (CVPR 2024 · similarity 0.88)
- In-Context Language Learning: Architectures and Algorithms (ICML 2024 · similarity 0.88)
- Vector-ICL: In-context Learning with Continuous Vector Representations (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.