Research map / Label-efficient and robust learning / Recognition and long-tail
Topic Modeling as Multi-Objective Contrastive Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes setwise contrastive learning for neural topic models with ChatGPT-based augmentation, combined with gradient-based multi-objective optimization (Pareto stationary solution) to balance the ELBO and contrastive losses.
Problem
Instance-wise contrastive learning in neural topic models captures low-level mutual information (e.g., word counts/ratios, peculiar words) and conflicts with the ELBO objective, and linear loss weighting is ineffective.
Approach
Contrast sets of documents (set-level representations pooled with min/max pooling) instead of individual pairs, with shuffled index matrices to increase set count, and balance the contrastive and ELBO objectives via a closed-form MGDA-style Pareto solver applied to encoder gradients.
Key results
- On 20NG (T=50), the model reaches NPMI 0.340 and TD 0.913, versus 0.332 and 0.853 for NTM+CL.
- On Wiki (T=50), NPMI 0.496 vs 0.481 for NTM+CL; TD 0.959 vs 0.841.
- Setwise MaxPooling for negatives and MinPooling for positives outperform other pooling functions such as GAP and FSP in ablations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use setwise contrastive learning with Pareto-based multi-objective optimization instead of instance-level contrastive training | Neural topic models · 20NG, IMDb, Wiki, and AG News; T=50 and T=200 | NPMI and TD: 20NG T=50 NPMI: 0.332 ± 0.006 -> 0.340 ± 0.005; TD: 0.853 ± 0.005 -> 0.913 ± 0.019; Wiki T=200 NPMI: 0.462 ± 0.006 -> 0. | Helped |
| Add Pareto stationary objective control rather than fixed linear or heuristic loss weighting (ablation) | Our model · 20NG and IMDb; T=50 and T=200 | NPMI and TD: 20NG T=50 NPMI: 0.322 ± 0.009 -> 0.340 ± 0.005; IMDb T=50 NPMI: 0.187 ± 0.005 -> 0.200 ± 0.007 | Helped |
| Replace Pareto solver with fixed linear weighting, α=0.5 (ablation) | Our model · 20NG and IMDb; T=50 and T=200 | NPMI: 20NG T=50: 0.340 ± 0.005 -> 0.322 ± 0.009; IMDb T=50: 0.200 ± 0.007 -> 0.187 ± 0.005 | Hurt |
| Replace negative-set MaxPooling with GAP (ablation) | Our model · 20NG, IMDb, Wiki, and AG News; T=50 and T=200 | NPMI: 20NG T=50: 0.340 ± 0.005 -> 0.339 ± 0.004; Wiki T=200: 0.491 ± 0.006 -> 0.485 ± 0.007 | Hurt |
| Use MaxPooling instead of MinPooling for positive-set representation (ablation) | Our model · 20NG, IMDb, Wiki, and AG News; T=50 and T=200 | NPMI: 20NG T=50: 0.333 ± 0.006 -> 0.340 ± 0.005; IMDb T=200: 0.191 ± 0.010 -> 0.197 ± 0.003 | Helped |
| Replace positive-set MinPooling with GAP (ablation) | Our model · 20NG, IMDb, Wiki, and AG News; T=50 and T=200 | NPMI: 20NG T=50: 0.340 ± 0.005 -> 0.336 ± 0.006; AG News T=200: 0.221 ± 0.003 -> 0.210 ± 0.002 | Hurt |
Surprising findings
- Linear combination of contrastive and ELBO losses (alpha 0.25/0.5) and existing multi-task weighting methods underperform the proposed Pareto solver.
- Instance-level contrastive NTMs can be swayed by peculiar words, e.g. inserting 'zeppelin' raises similarity of unrelated documents.
Limitations
- Relies on ChatGPT API for augmentation, adding cost and dependence on an external LLM.
- Gains over NTM+CL are modest on several datasets (e.g., NPMI differences of about 0.01-0.02).
Experimental setup
Datasets: 20Newsgroups IMDb Wikitext-103 AG News Yahoo Answers
Benchmarks: 20Newsgroups IMDb Wikitext-103 AG News Yahoo Answers
Metrics: NPMI topic diversity F1 human ratings word intrusion
Baselines: NTM ETM DVAE BATM W-LDA SCHOLAR SCHOLAR + BAT NTM+CL HyperMiner WeTe TSCTM UW
Builds on: NTM+CL Neural Topic Model (VAE) MGDA (gradient-based multi-objective optimization) InfoNCE PCGrad
Closest papers by meaning
- Contrastive Learning for Neural Topic Model (NeurIPS 2021 · similarity 0.87)
- Meta-Complementing the Semantics of Short Texts in Neural Topic Models (NeurIPS 2022 · similarity 0.86)
- Neural Dynamic Focused Topic Model (AAAI 2023 · similarity 0.85)
- Neural Topic Model via Optimal Transport (ICLR 2021 · similarity 0.85)
- Representing Mixtures of Word Embeddings with Mixtures of Topic Embeddings (ICLR 2022 · similarity 0.85)
- On the Affinity, Rationality, and Diversity of Hierarchical Topic Modeling (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.