Research map / Large language models / Language, safety and interpretability

Vector-ICL: In-context Learning with Continuous Vector Representations

ICLR 2025 · Large language models › Language, safety and interpretability · approach: Pruning and circuits in LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that LLMs can perform in-context learning on continuous vectors from black-box encoders via lightweight projectors (Vector-ICL), pretrained with language modeling and optionally finetuned per task.

Problem

Whether LLMs can perform in-context learning over continuous vector representations from diverse domains and encoders, rather than only discrete text tokens.

Approach

Project encoder outputs (from black-box pretrained encoders) into the LLM's embedding space with a lightweight linear or MLP projector ("box tokens"), pretrain the projector with next-token language modeling objectives while the encoder and LLM stay frozen, then optionally finetune the projector per task and use the projected vectors as in-context demonstrations.

Key results

• On text classification, finetuned Vector-ICL reaches 98.16 on SST2 and 97.28 on IMDB, above few-shot ICL (91.74, 93.50) but below soft prompt on Rotten Tomatoes (88.80 vs 93.24).
• On summarization, finetuned V-ICL gets 20.08 RougeL on XSum and 20.49 on XLSum, exceeding few-shot ICL (19.53, 19.41) and soft prompt (12.84, 12.70).
• Vector-ICL is reported to outperform the finetuned domain-specific baseline on time-series and graph classification, averaged over datasets and LLMs.

Measured effects reported in the paper

Surprising findings

• Pretraining projectors only with next-token language modeling (no task labels) already lets LLMs decode useful signals from noisy fMRI embeddings.
• Soft prompt tuning outperforms finetuned Vector-ICL on Rotten Tomatoes and remains strong on several classification datasets.

Limitations

• Only single-token encoders were tested; variable-length embeddings were not explored.
• Did not cover all combinations of encoders, LLM architectures, modalities, and tasks.
• Instruction tuning effects on vector-context understanding were not analyzed.

Experimental setup

Datasets: WikiText-103 Quora Parallel Sentence Talks 10-digit synthetic numbers IMDB Rotten Tomatoes SST2 Emotion Financial Phrasebank XSum XLSum Language + Molecules-24

Benchmarks: Quora Parallel Sentence Talks Rotten Tomatoes SST2 IMDB Emotion Financial Phrasebank XSum XLSum Language + Molecules-24 FordA FordB

Metrics: BLEU accuracy RougeL mean relative error F1

Baselines: few-shot ICL soft prompt tuning Graphormer finetuned Chronos-base classification head random baseline

Builds on: soft prompt tuning Chronos Graphormer Llama

Closest papers by meaning

• Making Text Embedders Few-Shot Learners (ICLR 2025 · similarity 0.90)
• In-Context Language Learning: Architectures and Algorithms (ICML 2024 · similarity 0.89)
• In-Context Learning State Vector with Inner and Momentum Optimization (NeurIPS 2024 · similarity 0.89)
• In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering (ICML 2024 · similarity 0.89)
• LIVE: Learnable In-Context Vector for Visual Question Answering (NeurIPS 2024 · similarity 0.89)
• Distributional Alignment as a Criterion for Designing Task Vectors in In-Context Learning (NeurIPS 2026 · similarity 0.89)