Research map / Multi-agent systems and games / LLM agents
Mem-PAL: Towards Memory-based Personalized Dialogue Assistants for Long-term User-Agent Interaction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PAL-Bench, a Chinese long-term user-agent personalization benchmark built from an LLM synthesis pipeline (PAL-Set), and H$^2$Memory, a hierarchical heterogeneous memory with RAG that outperforms prior memory baselines.
Problem
Personalized service-oriented dialogue assistants need to model user requirements and subjective preferences over long-term, multi-session interactions that include behavioral logs, which existing benchmarks and memory methods largely ignore.
Approach
Separately encode heterogeneous histories (behavior logs and dialogues) into concrete-level memories (log situation graph, dialogue topic outlines) and abstract-level memories (user background, requirement-preference principles), then retrieve across all four parts for RAG-based personalized response generation.
Key results
- H$^2$Memory achieves BLEU-1 26.67 and GPT-4 Score 32.54 on requirement restatement, beating the best baseline MemoryBank (28.57).
- On solution proposal, H$^2$Memory reaches Selection Score 38.32 vs 29.85 for MemoryBank.
- On LongMemEval single-session-preference, M_T + M_P reaches 50.00 accuracy vs 40.00 for ConditionMem.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add full H^2Memory hierarchical heterogeneous memory with RAG vs. vanilla with log | Qwen-Max-0428 response generation · PAL-Bench Solution Proposal | S-Score: 38.32 vs 22.88 | Helped |
| remove topic outline memory M_T (ablation) | H2Memory · PAL-Bench Solution Proposal | S-Score: -10.23 when removed (38.32 to 28.09) | Helped |
| remove topic outline memory M_T (ablation) | H2Memory · PAL-Bench Requirement Restatement | G-Score: -4.54 when removed (32.54 to 28.00) | Helped |
| add situation log graph memory M_G (ablation) | H2Memory · PAL-Bench Requirement Restatement | G-Score: +1.64 (30.90 to 32.54) | Helped |
| add situation log graph memory M_G (ablation) | H2Memory · PAL-Bench Solution Proposal | S-Score: +0.61 (37.71 to 38.32) | Helped |
| add background memory M_B (ablation) | H2Memory · PAL-Bench Solution Proposal | S-Score: +1.63 (36.69 to 38.32) | Helped |
| add background memory M_B (ablation) | H2Memory · PAL-Bench Requirement Restatement | G-Score: +1.24 (31.30 to 32.54) | Helped |
| add principle memory M_P (ablation) | H2Memory · PAL-Bench Solution Proposal | S-Score: +2.06 (36.26 to 38.32) | Helped |
| add principle memory M_P (ablation) | H2Memory · PAL-Bench Requirement Restatement | G-Score: +1.03 (31.51 to 32.54) | Helped |
| add principle memory M_P to dialogue memory M_T (ablation) | H2Memory on LongMemEval single-session-preference · LongMemEval single-session-preference | Accuracy: +10.00 (40.00 to 50.00) | Helped |
| add full H^2Memory memory-based RAG vs. vanilla (no memory) | Qwen-Max-0428 on LongMemEval single-session-preference · LongMemEval single-session-preference | Accuracy: +40.00 (10.00 to 50.00) | Helped |
| Multi-turn dialogue: H2Memory vs. memory baselines (win/tie/lose on requirement and preference) | H2Memory · PAL-Bench Multi-turn Dialogue Interaction | Win count vs. Vanilla (w/o log), preference: 480 wins / 18 ties / 328 losses | Helped |
Surprising findings
- Removing the topic-outline memory M_T causes the largest drop, while removing M_B hurts solution proposal more than removing M_G, reversing the trend seen for requirement restatement.
- Including logs in vanilla prompting improves performance, but the pipeline still shows non-negligible Lose cases against the vanilla baseline in dialogue evaluation.
Limitations
- Benchmark is fully synthetic (LLM-generated) with only 100 users and Chinese-language, with human evaluation on a small sample (5 users, 50 sessions).
- Multi-turn evaluation relies on LLM-as-judge (GPT-4-turbo) with positional bias mitigation, and the User-LLM has randomness that affects evaluation.
- Evaluated with a closed API model (Qwen-Max) and only a single external dataset subset for generalization.
Experimental setup
Datasets: PAL-Set LongMemEval
Benchmarks: PAL-Bench LongMemEval
Metrics: BLEU GPT-4 Score Selection Score Win/Tie/Lose Accuracy
Baselines: Vanilla (w/o log) Vanilla (with log) Turn-level RAG Session-level RAG RecurSum ConditionMem MemoryBank
Builds on: RecurSum MemoryBank ConditionMem
Closest papers by meaning
- AMemGym: Interactive Memory Benchmarking for Assistants in Long-Horizon Conversations (ICLR 2026 · similarity 0.91)
- LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (ICLR 2025 · similarity 0.89)
- Personalize-then-Store: Benchmarking and Learning Personalized Memory for Long-Horizon Agents (NeurIPS 2026 · similarity 0.89)
- PersonaVLM: Long-Term Personalized Multimodal LLMs (CVPR 2026 · similarity 0.89)
- MemArena: An Ego-Centric Benchmark for On-Device Agentic Personal Memory Assistants at Scale (NeurIPS 2026 · similarity 0.87)
- Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized Alignment (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.