Research map / Large language models / Language, safety and interpretability
LLM vs Small Model? Large Language Model Based Text Augmentation Enhanced Personality Detection Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills ChatGPT-generated semantic, sentiment, and linguistic post analyses and label descriptions into a BERT-based personality detector via contrastive learning and soft labels.
Problem
Personality detection from social media posts suffers from scarce ground-truth MBTI labels, leading to weak post representations, and treats traits as one-hot labels ignoring their semantics; LLMs perform poorly on the task directly.
Approach
Use an LLM to generate semantic, sentiment, and linguistic analyses of posts as positive samples for contrastive learning of a small BERT post encoder, and use LLM-generated label descriptions to compute soft labels, so no LLM is needed at inference.
Key results
- TAE achieves average Macro-F1 of 72.07% on Kaggle and 63.05% on Pandora, improving over the best baseline D-DGCN+l0 (71.35%) and D-DGCN (62.01%) by about 0.7 and 1.0 points respectively (text claims 1.01% and 1.68%).
- Zero-shot ChatGPT reaches 66.89% average Macro-F1 on Kaggle, comparable to fine-tuned BERT-mean (66.24%) but below TAE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TAE instead of D-DGCN+ℓ₀ | personality detection · Kaggle test | average Macro-F1 (%): 71.35 -> 72.07 | Helped |
| use TAE instead of D-DGCN | personality detection · Pandora test | average Macro-F1 (%): 62.01 -> 63.05 | Helped |
| add semantic post augmentation (ablation) | TAE · Kaggle test | average Macro-F1 (%): 71.40 -> 72.07 | Helped |
| add sentiment post augmentation (ablation) | TAE · Kaggle test | average Macro-F1 (%): 71.25 -> 72.07 | Helped |
| add linguistic post augmentation (ablation) | TAE · Kaggle test | average Macro-F1 (%): 71.07 -> 72.07 | Helped |
| add LLM-based label information enrichment (ablation) | TAE · Kaggle test | average Macro-F1 (%): 72.02 -> 72.07 | Helped |
| add TAE augmentations and label enrichment to original BERT with mean pooling (ablation) | personality detection · Kaggle test | average Macro-F1 (%): 66.24 -> 72.07 | Helped |
| replace contrastive learning with concatenation of post and analysis embeddings (ablation) | TAE · Kaggle test | average Macro-F1 (%): 72.07 -> 67.91 | Hurt |
| replace contrastive learning with weighted-sum pooling (ablation) | TAE · Kaggle test | average Macro-F1 (%): 72.07 -> 67.31 | Hurt |
| use chain-of-thought prompting instead of zero-shot prompting | ChatGPT personality detection · Kaggle test | Macro-F1 (%): 66.89 -> 65.12 | Hurt |
| use 3-shot prompting instead of zero-shot prompting | ChatGPT personality detection · Kaggle test | Macro-F1 (%): 66.89 -> 67.74 | Helped |
| increase the contrastive-loss trade-off parameter λ above 1 (ablation) | TAE · Kaggle and Pandora validation sets | Macro-F1 | Hurt |
Surprising findings
- ChatGPT with chain-of-thought (65.12%) and few-shot (67.74%) prompting did not improve over zero-shot performance on Kaggle.
- Semantic augmentation was the least influential of the three analysis aspects in ablation.
Limitations
- Relies on a single closed LLM (gpt-3.5-turbo-0301) for augmentation generation, cost-limited choice.
- Evaluated only on two MBTI datasets with Macro-F1 metric.
Experimental setup
Datasets: Kaggle MBTI Pandora
Benchmarks: Kaggle MBTI Pandora
Metrics: Macro-F1
Baselines: SVM XGBoost BiLSTM BERT-concat BERT-mean AttRCNN SN+Attn Transformer-MD TrigNet D-DGCN ChatGPT
Builds on: SimCSE CLAIF Transformer-MD D-DGCN TrigNet
Closest papers by meaning
- Leveraging the Dual Capabilities of LLM: LLM-Enhanced Text Mapping Model for Personality Detection (AAAI 2025 · similarity 0.94)
- Ask, Answer, and Detect: Role-Playing LLMs for Personality Detection with Question-Conditioned Mixture-of-Experts (NeurIPS 2026 · similarity 0.89)
- Data Augmented Graph Neural Networks for Personality Detection (AAAI 2024 · similarity 0.87)
- Neuron based Personality Trait Induction in Large Language Models (ICLR 2025 · similarity 0.85)
- Personality Alignment of Large Language Models (ICLR 2025 · similarity 0.85)
- Knowledge-Enhanced Hierarchical Heterogeneous Graph for Personality Identification with Limited Training Data (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.