Research map / Probabilistic and scientific modeling / Causal discovery
E-LDA: Toward Interpretable LDA Topic Models with Strong Guarantees in Logarithmic Parallel Time
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes E-LDA, a combinatorial submodular reformulation of LDA MAP topic-word assignment under a sparsity constraint, solved with fast greedy and logarithmic-adaptivity parallel algorithms, with topics generated from keyword-anchored coherence functions.
Problem
LDA inference of per-document topic assignments lacks provable guarantees, is incompatible with out-of-sample causal inference, and yields uninterpretable topics.
Approach
Reformulate LDA MAP topic-word assignment (in the alpha-to-infinity limit) as maximizing a monotone submodular set function over topic-document links under a cardinality constraint, then solve it with a fast greedy algorithm and a logarithmic-adaptivity parallel algorithm, using candidate topics built from keyword-anchored coherence similarities.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Gibbs assignment inference with E-LDA while keeping Gibbs topics and matching assignment sparsity | LDA assignments using Gibbs topics · Congress, Reuters, and NewsGroups | log posterior: Congress: -2.05 -> -1.78; Reuters: -2.80 -> -2.52; NewsGroups: -1.57 -> -1.40 | Helped |
| Replace Gibbs-Bayes assignment inference with E-LDA while keeping Gibbs-Bayes topics and matching assignment sparsity | LDA assignments using Gibbs-Bayes topics · Congress, Reuters, and NewsGroups | log posterior: Congress: -2.47 -> -2.22; Reuters: -3.66 -> -3.43; NewsGroups: -1.70 -> -1.61 | Helped |
| Replace Gibbs-Sparse assignment inference with E-LDA while keeping Gibbs-Sparse topics and matching assignment sparsity | LDA assignments using Gibbs-Sparse topics · Congress, Reuters, and NewsGroups | log posterior: Congress: -2.43 -> -2.14; Reuters: -3.49 -> -3.13; NewsGroups: -1.68 -> -1.55 | Helped |
| Use the co-occurrence topic generator instead of Gibbs topics (ablation) | E-LDA · Text datasets; top 5 words per topic | UMass coherence: 2.53-4.20 times better mean coherence | Helped |
| Use the Exp-UMass topic generator instead of Gibbs topics (ablation) | E-LDA · Text datasets; top 5 words per topic | UMass coherence: 1.18-1.67 times better mean coherence | Helped |
| Measure coherence over 10-25 top words rather than 5 top words (ablation) | E-LDA co-occurrence and Exp-UMass topics · Text datasets | UMass coherence relative to Gibbs: co-occurrence: 2.92-3.21; exp-UMass: 1.93-2.33 | Helped |
| Use the co-occurrence topic generator instead of neural and LLM-based topic models (ablation) | E-LDA · Text datasets | UMass coherence: BERTopic: 3.04-4.70; AVITM: 5.96-10.17; FASTopic: 2.24-5.84; ETM: 1.19-2.36 | Helped |
| Use the Exp-UMass topic generator instead of neural and LLM-based topic models (ablation) | E-LDA · Text datasets | UMass coherence: BERTopic: 1.46-2.72; AVITM: 3.08-7.09; FASTopic: 1.48-2.67 | Mixed |
| Use the unmodified UMass topic generator instead of co-occurrence or Exp-UMass (ablation) | E-LDA · Text datasets | UMass coherence | Hurt |
| Vary solution sparsity from 1 to 10 topics per document (ablation) | E-LDA topic solutions · Text datasets | UMass coherence | No clear effect |
Surprising findings
- The unmodified UMass topic generator performs relatively poorly within E-LDA despite UMass being the coherence metric, showing the generator's form matters.
Limitations
- Results rely on the alpha-to-infinity limit and a sparsity constraint rather than a learned prior.
- Evaluated mainly on UMass coherence, which the generators are similar to.
Experimental setup
Datasets: Reuters news articles US Congressional speeches political news discussion board posts (NewsGroups)
Benchmarks: Reuters Congress NewsGroups
Metrics: log posterior of assignments UMass topic coherence
Baselines: MALLET Gibbs LDA Gibbs-Bayes Gibbs-Sparse BERTopic AVITM ETM FASTopic
Builds on: LDA FAST parallel submodular maximization UMass coherence
Closest papers by meaning
- Discriminative Topic Modeling with Logistic LDA (NeurIPS 2019 · similarity 0.88)
- A Reduction for Efficient LDA Topic Reconstruction (NeurIPS 2018 · similarity 0.88)
- Provable Algorithms for Inference in Topic Models (ICML 2016 · similarity 0.87)
- Autoencoding Variational Inference For Topic Models (ICLR 2017 · similarity 0.87)
- Online Bayesian Moment Matching for Topic Modeling with Unknown Number of Topics (NeurIPS 2016 · similarity 0.86)
- Knowledge-Aware Bayesian Deep Topic Model (NeurIPS 2022 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.