Research map / Architectures and efficiency / Attention and transformers
FASTopic: Pretrained Transformer is a Fast, Adaptive, Stable, and Transferable Topic Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FASTopic, a fast topic model that reconstructs document-word relations via pretrained Transformer document embeddings and learnable topic/word embeddings, regularized with entropic optimal transport plans (ETP).
Problem
Existing topic models lack efficiency, effectiveness, or stability: VAE-based models are slow and complex, clustering-based models yield repetitive topics and poor doc-topic distributions, and neural models are hyperparameter-sensitive.
Approach
Dual Semantic-relation Reconstruction (DSR) models doc-topic and topic-word distributions as relations between document, topic and word embeddings and reconstructs the BoW through them; Embedding Transport Plan (ETP) regularizes these relations as entropic optimal transport plans (Sinkhorn) with learnable topic and word weights to avoid the relation bias of parameterized softmax.
Key results
- FASTopic achieves C_V 0.426 and TD 0.983 on 20NG and the best topic quality on all six datasets in the reported table, versus ECRTM as the strongest baseline.
- FASTopic runs in 10.6s on 20NG versus 365.4s for ECRTM and 58.6s for LDA-Mallet, and completes all datasets within about 1 minute (12.5s on NYT, 60.3s on ACL).
- Ablation replacing ETP with parameterized softmax drops 20NG C_V from 0.426 to 0.368 and TD from 0.983 to 0.391.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ETP instead of parameterized softmax for semantic relations (ablation) | FASTopic · 20NG | TD: 0.391 -> 0.983 | Helped |
| Use ETP instead of parameterized softmax for semantic relations (ablation) | FASTopic · 20NG | C_V: 0.368 -> 0.426 | Helped |
| Use ETP instead of parameterized softmax for semantic relations (ablation) | FASTopic · 20NG | Purity: 0.401 -> 0.577 | Helped |
| Use ETP instead of parameterized softmax for semantic relations (ablation) | FASTopic · 20NG | NMI: 0.452 -> 0.525 | Helped |
| Use ETP instead of parameterized softmax for semantic relations (ablation) | FASTopic · NYT | C_V: 0.363 -> 0.437 | Helped |
| Use ETP instead of parameterized softmax for semantic relations (ablation) | FASTopic · NYT | TD: 0.338 -> 0.999 | Helped |
| Use ETP instead of parameterized softmax for semantic relations (ablation) | FASTopic · WoS | C_V: 0.395 -> 0.457 | Helped |
| Use ETP instead of parameterized softmax for semantic relations (ablation) | FASTopic · WoS | TD: 0.522 -> 1.000 | Helped |
| Use ETP instead of parameterized softmax for semantic relations (ablation) | FASTopic · NeurIPS | C_V: 0.380 -> 0.422 | Helped |
| Use ETP instead of parameterized softmax for semantic relations (ablation) | FASTopic · NeurIPS | TD: 0.216 -> 0.998 | Helped |
| Use ETP instead of parameterized softmax for semantic relations (ablation) | FASTopic · ACL | C_V: 0.373 -> 0.420 | Helped |
| Use ETP instead of parameterized softmax for semantic relations (ablation) | FASTopic · ACL | TD: 0.164 -> 0.998 | Helped |
Surprising findings
- Parameterized softmax suffers a relation bias issue where most topic embeddings collapse together, causing repetitive topics even with only 10 topics.
- Using pretrained word embeddings like word2vec or GloVe is avoided because they are not in the same semantic space as document embeddings.
Limitations
- Bounded by the maximum input length of pretrained document embedding models, which may hurt extremely long documents.
- Relies on the quality of the pretrained document embedding model; performance varies with the embedding model choice.
Experimental setup
Datasets: 20NG NYT WoS NeurIPS ACL Wikitext-103
Benchmarks: 20NG NYT WoS NeurIPS ACL Wikitext-103
Metrics: C_V topic diversity Purity NMI running time classification accuracy F1
Baselines: LDA-Mallet NMF BERTopic CombinedTM GINopic ProGBN HyperMiner ECRTM
Builds on: ECRTM Sinkhorn algorithm Sentence-BERT
Closest papers by meaning
- Context-guided Embedding Adaptation for Effective Topic Modeling in Low-Resource Regimes (NeurIPS 2023 · similarity 0.88)
- Neural Topic Model via Optimal Transport (ICLR 2021 · similarity 0.87)
- OTLDA: A Geometry-aware Optimal Transport Approach for Topic Modeling (NeurIPS 2020 · similarity 0.86)
- Sparse Autoencoders are Topic Models (ICML 2026 · similarity 0.86)
- How Do Transformers Learn Topic Structure: Towards a Mechanistic Understanding (ICML 2023 · similarity 0.86)
- On the Affinity, Rationality, and Diversity of Hierarchical Topic Modeling (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.