Research map / Architectures and efficiency / Attention and transformers
Modeling Language Tokens as Functionals of Semantic Fields
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LasF, a Transformer-free nonlinear module that models language tokens as functionals over semantic fields, used both to replace final layers of pretrained LMs and as a flat from-scratch language model.
Problem
Transformers are parameter-heavy and deep, while existing Transformer-free models (SSMs) rely on linear token relations and lack a neuro-biological connection to how the brain represents semantic relations.
Approach
Represent tokens via neuronal states in multiple semantic fields and model interactions between fields as a functional (field-level metric matrix weighting neuron-level relational matrices), yielding a nonlinear module that replaces final Transformer layers or whole stacks, enabling flat (shallow) models.
Key results
- ALBERT+LasF reaches 82.2% single-model CommonsenseQA blind test accuracy, 3.1% above UnifiedQA (11B) and 3.8% above ALBERT+HeadHunter.
- A 2-layer LasF language model reaches perplexity 17.2 on WikiText-103 and 49.2 on Penn Treebank, versus 45.8 and 72.5 for a 2-layer Transformer.
- LasF on LLaMA-13B with 86.1M head parameters raises CSQA accuracy from 67.3% to 69.4% and PIQA from 80.1% to 82.1%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the output layer with LasF | ALBERT-base · SQuAD2.0 | F1: 79.15 -> 80.33 | Helped |
| Replace the output layer with LasF | RoBERTa-base · SQuAD2.0 | F1: 82.32 -> 83.35 | Helped |
| Replace the output layer with LasF | DeBERTa-base · SQuAD2.0 | F1: 84.50 -> 85.54 | Helped |
| Replace the output layer with LasF | LLaMA-7B · CSQA | accuracy: 57.8 -> 62.2 | Helped |
| Replace the output layer with LasF | LLaMA-13B · PIQA | accuracy: 80.1 -> 82.1 | Helped |
| Replace the output layer with LasF | LLaMA-13B · SIQA | accuracy: 50.4 -> 51.8 | Helped |
| Use L1-norm instead of the default neuron-level metric µ (ablation) | 2-layer LasF · WikiText-103 | perplexity: 17.2 -> 17.6 | Hurt |
| Use L1-norm instead of the default neuron-level metric µ (ablation) | 2-layer LasF · PennTreebank | perplexity: 49.2 -> 49.5 | Hurt |
| Use only L1-norm in the field-level metric set µk (ablation) | 2-layer LasF · WikiText-103 | perplexity: 17.2 -> 18.3 | Hurt |
| Use only L1-norm in the field-level metric set µk (ablation) | 2-layer LasF · PennTreebank | perplexity: 49.2 -> 52.2 | Hurt |
| Use truncated length-related projections D and DS (ablation) | 2-layer LasF · WikiText-103 | perplexity: 17.2 -> 17.6 | Hurt |
| Use truncated length-related projections D and DS (ablation) | 2-layer LasF · PennTreebank | perplexity: 49.2 -> 49.3 | No clear effect |
Surprising findings
- A flat language model with only 1 to 3 LasF layers outperforms 10+ layer Transformer and SSM baselines on WikiText-103 perplexity.
- LasF replacing the head of LLaMA-7B slightly lowers CSQA accuracy (57.8% to 56.9%) in one configuration.
Limitations
- Lacks well-established CUDA kernels for metric-based operations, so efficiency gains are not yet fully realized.
- Hyper-parameter tuning is sensitive since the neuronal-group concepts are underexplored.
- Language modeling experiments are small-scale and cannot conclusively show LasF beats Transformers at scale.
Experimental setup
Datasets: SQuAD2.0 CommonsenseQA PIQA SIQA WikiText-103 Penn Treebank WikiQA
Benchmarks: SQuAD2.0 CommonsenseQA PIQA SIQA WikiText-103 Penn Treebank
Metrics: EM F1 accuracy perplexity parameter count training and inference time
Baselines: RoBERTa-large ALBERT-xxlarge UnifiedQA ALBERT+HeadHunter QA-GNN S4 H3 Hyena Transformer GPT-2 small Reformer
Builds on: state-space models Transformer Pei & Wang (2023)
Closest papers by meaning
- Crafting Interpretable Embeddings for Language Neuroscience by Asking LLMs Questions (NeurIPS 2024 · similarity 0.85)
- Improving Semantic Understanding in Speech Language Models via Brain-tuning (ICLR 2025 · similarity 0.85)
- Language Models Are Implicitly Continuous (ICLR 2025 · similarity 0.85)
- The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction (ICLR 2024 · similarity 0.84)
- PaceLLM: Brain-Inspired Large Language Models for Long-Context Understanding (NeurIPS 2025 · similarity 0.84)
- Talking Heads: Understanding Inter-Layer Communication in Transformer Language Models (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.