Research map / Graph learning / Graph self-supervised learning
ICLR: In-Context Learning of Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that LLMs reorganize intermediate representations of pretrained concepts into an in-context graph structure as context grows, explained via Dirichlet energy minimization and spectral embeddings.
Problem
Whether LLMs can override pretraining semantics of concepts and reorganize their internal representations according to semantics specified only in-context.
Approach
Uses an in-context graph tracing task (random walks on grids, rings, hex lattices with concept tokens as nodes) and shows representations sharply reorganize into graph geometry as context scales; hypothesizes this is implicit Dirichlet energy minimization whose minimizers are spectral embeddings matching PCA components.
Key results
- Llama-3.1-8B accuracy on in-context graph tracing rises sharply after a critical context length, with Dirichlet energy minimized shortly before the accuracy jump.
- Rule-following accuracy of memorization baselines (1-shot, 2-shot) fails to explain Llama's ascent curve on a 50-node ring and 25-node grid, which shows a two-phase (slow then fast) pattern.
- Under a conflicting semantic prior (shuffled days of the week), the in-context ring appears in PCs 3-4 while the semantic ring stays in PCs 1-2, and the model's accuracy shifts to the in-context rule as context grows.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Rescale 3 principal components toward a target node instead of applying a null intervention | Llama-3.1-8B · Ring | Hit@1: 0.20 -> 0.77 | Helped |
| Rescale 3 principal components toward a target node instead of applying a null intervention | Llama-3.1-8B · Ring | Hit@3: 0.50 -> 0.96 | Helped |
| Rescale 3 principal components toward a target node instead of applying a null intervention | Llama-3.1-8B · Ring | Prob: 0.20 -> 0.76 | Helped |
| Rescale 3 principal components toward a target node instead of applying a null intervention | Llama-3.1-8B · Grid | Hit@1: 0.17 -> 0.68 | Helped |
| Rescale 3 principal components toward a target node instead of applying a null intervention | Llama-3.1-8B · Grid | Hit@3: 0.33 -> 0.98 | Helped |
| Rescale 3 principal components toward a target node instead of applying a null intervention | Llama-3.1-8B · Grid | Prob: 0.16 -> 0.65 | Helped |
| Rescale 3 principal components toward a target node instead of applying a null intervention | Llama-3.1-8B · Hex | Hit@1: 0.07 -> 0.42 | Helped |
| Rescale 3 principal components toward a target node instead of applying a null intervention | Llama-3.1-8B · Hex | Hit@3: 0.20 -> 0.46 | Helped |
| Rescale 3 principal components toward a target node instead of applying a null intervention | Llama-3.1-8B · Hex | Prob: 0.05 -> 0.82 | Helped |
Surprising findings
- Context-specified structure is present in the representations but cannot dominate the pretrained semantic structure, appearing only in higher principal components.
- Hexagonal graph tracking accuracy of Llama-3.1-8B is lower than the 1,2-shot memorization baseline, suggesting a different underlying process.
Limitations
- PCA visualizations can be difficult to interpret or misleading; causal interventions on principal components only partially explain model predictions.
- Graph sizes are small, so finite-size effects may dominate the percolation-style transition analogy.
- Dirichlet energy minimization hypothesis is supported by correlation with accuracy rather than a full mechanistic account.
Experimental setup
Datasets: synthetic graph random-walk sequences over concept tokens
Benchmarks: in-context graph tracing (synthetic)
Metrics: rule-following accuracy Dirichlet energy Hit@1 Hit@3 accumulated probability mass on valid neighbors
Baselines: 1-shot memorization solution 2-shot memorization solution
Builds on: linear representation hypothesis in-context vector steering in-context learning as implicit optimization spectral graph theory
Closest papers by meaning
- Attention Mechanisms Perspective: Exploring LLM Processing of Graph-Structured Data (ICML 2025 · similarity 0.88)
- How Do Large Language Models Understand Graph Patterns? A Benchmark for Graph Pattern Comprehension (ICLR 2025 · similarity 0.88)
- The Representation Landscape of Few-Shot Learning and Fine-Tuning in Large Language Models (NeurIPS 2024 · similarity 0.88)
- Talk like a Graph: Encoding Graphs for Large Language Models (ICLR 2024 · similarity 0.87)
- Can Large Language Models Analyze Graphs like Professionals? A Benchmark, Datasets and Models (NeurIPS 2024 · similarity 0.87)
- One For All: Towards Training One Graph Model For All Classification Tasks (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.