Research map / Graph learning / Graph self-supervised learning

Improving LLM Final Representations with Inter-Layer Geometry

NeurIPS 2026 · Graph learning › Graph self-supervised learning · approach: Multi-view graph clustering · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Cayley-Encoder, a GNN over a sparse Cayley graph on SL(2,Z_n) whose nodes are frozen-LLM layer representations, to aggregate inter-layer signals for downstream prediction.

Problem

Final-layer representations of LLMs may miss task-relevant signals in intermediate layers, while existing fixes either pick the best layer per task or use costly attention-based inter-layer aggregation.

Approach

Treat each LLM layer's mean-pooled representation as a node in a fixed graph and aggregate with a GNN; a 4-regular Cayley graph over SL(2,Z_n) gives a sparse, regular, low-diameter topology that is cheaper than a fully connected layer graph while keeping inter-layer communication effective.

Key results

• Cayley-Encoder achieves best score in 10 of 15 classification configurations and 17 of 24 STS configurations across Pythia-410M, Gemma2-2B, and Llama3-8B.
• Gains of up to 40 percentage points over Last-Layer on the Emotion task; average 29 pp over Last-Layer on classification.
• Cayley-Encoder outperforms LoRA (frozen LLM) on 13 of 15 classification and 21 of 24 STS configurations, with at most 0.1% added parameters.

Surprising findings

• Adding learnable layer-position encodings generally does not help and often degrades performance.
• Layer-to-node permutation has little effect on accuracy (std below 2 pp), suggesting layer identity is not a useful signal.

Limitations

• Evaluated only on small-to-medium LLMs (up to 8B) and classification/STS tasks; generative settings not explored.
• Results are single-run per configuration following MTEB protocol, with no variance reported for main tables.
• Cayley graph sizes are fixed by the group, requiring virtual nodes that may waste capacity.

Experimental setup

Datasets: Banking77 Emotion MTOP Domain MTOP Intent Poem Sentiment STSBenchmark STS12 STS13 STS14 STS15 STS16 BIOSSES

Benchmarks: Banking77 Emotion MTOP Domain MTOP Intent Poem Sentiment STSBenchmark STS12 STS13 STS14 STS15 STS16 BIOSSES

Metrics: accuracy Spearman correlation

Baselines: Last-Layer Best-Layer Weighted (ELMo-style) MLP Last-Layer DWAtt Deepset LoRA Random-Regular graph

Builds on: Expander Graph Propagation Cayley Graph Propagation DWAtt ELMo LoRA

Closest papers by meaning

• LLMs as Zero-shot Graph Learners: Alignment of GNN Representations with LLM Token Embeddings (NeurIPS 2024 · similarity 0.86)
• Dynamics of the Transformer Residual Stream: Coupling Spectral Geometry to Network Topology (NeurIPS 2026 · similarity 0.84)
• Neural Snowflakes: Universal Latent Graph Inference via Trainable Latent Geometries (ICLR 2024 · similarity 0.84)
• TopoTune: A Framework for Generalized Combinatorial Complex Neural Networks (ICML 2025 · similarity 0.84)
• Bootstrapping Heterogeneous Graph Representation Learning via Large Language Models: A Generalized Approach (AAAI 2025 · similarity 0.84)
• A Unified Graph Language Model for Multi-Domain Multi-Task Graph Alignment Instruction Tuning (NeurIPS 2026 · similarity 0.84)