Research map / Large language models / Language, safety and interpretability
PANTHER: Generative Pretraining Beyond Language for Sequential User Behavior Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends generative pretraining to user payment behavior sequences with structured tokenization, a pattern-aware SPRM module, and contrastive user-profile embeddings, deployed for fraud detection at WeChat Pay and evaluated on public transaction and recommendation benchmarks.
Problem
Discriminative industrial models for payment behavior sequences suffer from label scarcity, overfitting to high-cardinality features, truncated long histories, and static embeddings; generative pretraining has not been applied to user behavior as a transferable modality.
Approach
Pretrain a transformer with a SPRM (depthwise dilated convolutions plus prototype cross-attention) on next-transaction prediction, add a user-profile token and contrastive loss over demographically similar users, then cache embeddings offline and fuse them with real-time features in a lightweight discriminative fraud model.
Key results
- On WeChat Pay next-transaction prediction, PANTHER with profile and contrastive learning reaches HR@1 0.2452 vs 0.1952 for Transformer (+25.6%).
- Hybrid PANTHER improves fraud recall at 0.1% threshold by 38.6% over DeepFM baseline in a 10-day online A/B test.
- On MovieLens-1M PANTHER improves HR@1 by 21.76% over Transformer (0.0705 vs 0.0579); on Yelp, NDCG@5 improves 29.6% over DCN in hybrid setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SPRM for next-transaction prediction rather than the Transformer baseline | PANTHER · WeChat Pay | HR@1: 0.1952 -> 0.2243; +14.9% | Helped |
| Add user profile as the first token (ablation) | PANTHER with SPRM · WeChat Pay | HR@1: 0.2243 -> 0.2301 | Helped |
| Add user profile as positional encoding (ablation) | PANTHER with profile as first token · WeChat Pay | HR@1: 0.2301 -> 0.2351 | Helped |
| Add contrastive learning to the profile-based model (ablation) | PANTHER with profile as positional encoding · WeChat Pay | HR@1: 0.2351 -> 0.2452 | Helped |
| Set the loss coefficient λ to 0.2 rather than 0.1 (ablation) | PANTHER · WeChat Pay | HR@1: 0.2452 -> 0.2441 | Hurt |
| Set the loss coefficient λ to 0.4 rather than 0.1 (ablation) | PANTHER · WeChat Pay | HR@1: 0.2452 -> 0.2435 | Hurt |
| Set the loss coefficient λ to 0.8 rather than 0.1 (ablation) | PANTHER · WeChat Pay | HR@1: 0.2452 -> 0.2430 | Hurt |
| Use PANTHER instead of the strongest listed baseline, HSTU | Next-behavior prediction · MBD-mini | HR@1: .0432 -> .0454; +2.95% vs Transformer | Helped |
| Use PANTHER instead of HSTU | Next-behavior prediction · CCT | HR@1: .0537 -> .0576 | Helped |
| Use PANTHER instead of HSTU | Next-item prediction · MovieLens-1M | HR@1: .0699 -> .0705 | Helped |
| Use PANTHER instead of HSTU | Next-item prediction · Yelp | HR@1: .0879 -> .0929 | Helped |
| Add PANTHER pretrained embeddings and predictions to DCN | DCN recommendation model · Yelp | NDCG@5: 0.534 -> 0.692 (+29.6%) | Helped |
Surprising findings
- SASRec has lower HR@1 than Transformer on CCT (-10.6%), and HSTU is worse than Transformer on MBD-mini at HR@1.
Limitations
- Limited interpretability of complex representations, acknowledged in conclusion.
- Proprietary WeChat Pay data is not released, so industrial results are not reproducible.
- Benchmark experiments use single-GPU training with small configurations (4 layers, 2 heads).
Experimental setup
Datasets: WeChat Pay transactions (proprietary) CCT MBD-mini MovieLens-1M Yelp
Benchmarks: WeChat Pay next-transaction prediction MBD-mini CCT MovieLens-1M Yelp
Metrics: HR@1 HR@5 HR@10 NDCG@5 recall F1 score accuracy
Baselines: Transformer SASRec HSTU DeepFM DCN TabBERT-MLP TabBERT-LSTM Cold-Start
Builds on: SASRec HSTU DCN DeepFM TextCNN
Closest papers by meaning
- BehaviorBench: Modeling Real-World User Decisions from Behavioral Traces (NeurIPS 2026 · similarity 0.88)
- Pre-trained Behavioral Model for Malicious User Prediction on Social Platform (AAAI 2025 · similarity 0.86)
- Towards Real-world Human Behavior Simulation: Benchmarking Large Language Models on Long-horizon, Cross-scenario, Heterogeneous Behavior Traces (NeurIPS 2026 · similarity 0.85)
- PICLe: Eliciting Diverse Behaviors from Large Language Models with Persona In-Context Learning (ICML 2024 · similarity 0.85)
- HYDRA: Model Factorization Framework for Black-Box LLM Personalization (NeurIPS 2024 · similarity 0.84)
- CogBench: a large language model walks into a psychology lab (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.