Research map / Vision-language and multimodal / Human and embodied modeling
The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the semantic hub hypothesis that LMs share a modality-agnostic representation space across languages, arithmetic, code, formal semantics, images and audio, anchored by the dominant language (usually English), and shows via logit lens and activation-steering interventions that this space is causally used.
Problem
How language models process heterogeneous data types (multiple languages, code, arithmetic, formal semantics, images, audio) within one set of parameters, and whether semantically equivalent inputs from different data types share a representation space.
Approach
Hypothesizes that a single multi-data-type LM learns a shared intermediate representation space (a "semantic hub") where semantically equivalent inputs cluster, which is anchored by dominant-language tokens and can be read out via the logit lens and intervened on cross-modally without any explicit alignment transformation.
Key results
- In Llama-3, English token probability under the logit lens exceeds the Chinese token probability after layer 17 until layer 31 when processing Chinese prefixes.
- Cross-lingual ActAdd steering with English trigger words achieves sentiment steering comparable to same-language steering for Llama-3 on Spanish and Chinese inputs.
- Arithmetic intervention in English steers Llama-3/Llama-2 outputs to the expected decremented result for over 90% of instances at sufficient intervention strength.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply English negative sentiment steering | Llama-3 processing Spanish text · Spanish sentiment steering | sentiment: 0.143$_{\pm0.022}$ -> 0.139$_{\pm0.026}$ | Hurt |
| Apply English positive sentiment steering | Llama-3 processing Spanish text · Spanish sentiment steering | sentiment: 0.143$_{\pm0.022}$ -> 0.159$_{\pm0.026}$ | Helped |
| Apply English negative sentiment steering | Llama-3 processing Chinese text · Chinese sentiment steering | sentiment: 0.178$_{\pm0.030}$ -> 0.161$_{\pm0.029}$ | Hurt |
| Apply English positive sentiment steering | Llama-3 processing Chinese text · Chinese sentiment steering | sentiment: 0.178$_{\pm0.030}$ -> 0.179$_{\pm0.032}$ | No clear effect |
| Apply English negative sentiment steering | Llama-2 processing Spanish text · Spanish sentiment steering | sentiment: 0.144$_{\pm0.014}$ -> 0.097$_{\pm0.024}$ | Hurt |
| Apply English positive sentiment steering | Llama-2 processing Spanish text · Spanish sentiment steering | sentiment: 0.144$_{\pm0.014}$ -> 0.149$_{\pm0.015}$ | Helped |
| Apply English negative sentiment steering | Llama-2 processing Chinese text · Chinese sentiment steering | sentiment: 0.223$_{\pm0.036}$ -> 0.156$_{\pm0.076}$ | Helped |
| Apply English positive sentiment steering | Llama-2 processing Chinese text · Chinese sentiment steering | sentiment: 0.223$_{\pm0.036}$ -> 0.227$_{\pm0.038}$ | No clear effect |
| Use English-token activation interventions to decrement an arithmetic result and elicit a +1 correction | Llama-2 and Llama-3 arithmetic generation · Arithmetic expressions | successful steering rate: up to $>90\%$ | Helped |
| Use English start/end trigger tokens to steer range(end) toward range(0, end) | Llama-2 code generation · MBPP range calls | successful steering rate: up to 67% | Helped |
| Replace image-patch representations of one color with another color's English-token representation | Chameleon color-question answering · Two-color image questions | successful steering rate: as high as above 80% | Helped |
| Add English animal-word intervention vectors in the steered direction | SALMONN animal-sound classification · VGGSound animal sounds | normalized Yes probability | Helped |
Surprising findings
- BLOOM, a multilingual model without a dominant language, shows no clear latent language in intermediate layers under the logit lens, though translation similarity still appears.
- Replacing image patch hidden states in Chameleon with English color word unembeddings steers its color answers in over 80% of cases at some layers.
Limitations
- Logit lens restricted to single-token verbalizations and tokenization mismatches require aggressive filtering.
- Some experiments rely on intervention vectors that may succeed due to cross-lingual embedding alignment, and multimodal analysis is limited without non-text vocabulary tokens.
- Results are limited to a small set of models and simple synthetic or narrow tasks.
Experimental setup
Datasets: GALE English-Chinese parallel sentences Wikipedia mC4 MBPP COGS MSCOCO 2017 VGGSound InterTASS multilingual Amazon reviews
Benchmarks: MBPP COGS MSCOCO 2017 VGGSound InterTASS multilingual Amazon reviews
Metrics: cosine similarity logit lens probability sentiment score perplexity (disfluency) relevance (cosine similarity) steering success rate Yes probability
Baselines: Llama-2 English-space finding (Wendler et al.) CLIP BLOOM
Builds on: logit lens Activation Addition (ActAdd) hub-and-spoke model of semantic memory
Closest papers by meaning
- The Emergence of Abstract Thought in Large Language Models Beyond Any Language (NeurIPS 2025 · similarity 0.89)
- Less is More: Local Intrinsic Dimensions of Contextual Language Models (NeurIPS 2025 · similarity 0.88)
- TopoLM: brain-like spatio-functional organization in a topographic language model (ICLR 2025 · similarity 0.87)
- Improving Semantic Understanding in Speech Language Models via Brain-tuning (ICLR 2025 · similarity 0.87)
- Far from the Shallow: Brain-Predictive Reasoning Embedding through Residual Disentanglement (NeurIPS 2025 · similarity 0.87)
- Brain-tuning Improves Generalizability and Efficiency of Brain Alignment in Speech Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.