Research map / Large language models / Agents, code and math
Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the LTM Benchmark, a synthetic single-conversation benchmark that interleaves multiple memory/continual-learning tasks to evaluate LLMs and LTM-augmented agents at memory spans up to 500k tokens.
Problem
Standard LLM evaluations are isolated, single-shot prompts that do not capture long, interleaved chat conversations requiring long-term memory, continual learning and information integration.
Approach
Evaluate agents through one long simulated user-agent conversation in which multiple tests are context-switched and interleaved with distractor messages, scored automatically (regex, string match, LLM-based checkers) across memory spans from 2k to 500k tokens.
Key results
- Claude 3 Opus scores 8.3 on isolated tasks but 7.8 at 2k and 3.4 at 500k; Gemini 1.5 Pro is the only model to reach 7.7 at 200k and 6.1 at 500k.
- GPT-4 turbo drops from 7.9 (isolated) to 6.6 (2k) and 1.0 (500k), while LTM GPT-4 turbo (16k context) scores 9.2 isolated and 3.1 at 500k.
- Open-source Mixtral 8x7B drops from 5.0 isolated to 1.4 at 2k and 0.1 at 32k, showing interleaving strongly degrades performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Interleave tasks instead of evaluating them in isolation | Llama 3 70B · LTM Benchmark | score: 8.2 -> 1.9 | Hurt |
| Interleave tasks instead of evaluating them in isolation | GPT-4 turbo · LTM Benchmark | score: 7.9 -> 6.6 | Hurt |
| Increase the memory span from 2k to 32k | GPT-4 turbo · LTM Benchmark | score: 6.6 -> 5.2 | Hurt |
| Increase the memory span from 2k to 32k | Llama 3 70B · LTM Benchmark | score: 1.9 -> 0.2 | Hurt |
| Use LTM with GPT-4 turbo rather than the vanilla model | GPT-4 turbo · LTM Benchmark isolated | score: 7.9 -> 9.2 | Helped |
| Use LTM with Llama 3 70B rather than the vanilla model | Llama 3 70B · LTM Benchmark 2k | score: 1.9 -> 6.9 | Helped |
| Use LTM with Claude 3 Opus rather than the vanilla model | Claude 3 Opus · LTM Benchmark 2k | score: 7.8 -> 7.5 | Hurt |
| Use LTM with GPT-4o-mini rather than the vanilla model | GPT-4o-mini · LTM Benchmark 500k | score: 1.4 -> 4.7 | Helped |
| Increase the memory span from 2k to 500k | Claude 3 Opus · LTM Benchmark | score: 7.8 -> 3.4 | Hurt |
Surprising findings
- GPT-4o scores similarly to GPT-4 turbo despite near-perfect NIAH and NIAN results.
- Short-context LLMs with an LTM system perform as well as or better than much larger-context models at long memory spans.
Limitations
- Only three repetitions per scenario, so results have high variance and limited reliability.
- Automated LLM-based evaluation sometimes fails, requiring manual correction.
- High time and financial cost of running the benchmark limits repeats.
Experimental setup
Datasets: TriviaQA ChapterBreak ToMi (ToM QA) PG-19 (via ChapterBreak)
Benchmarks: LTM Benchmark
Metrics: score out of 11 per configuration standard deviation across repetitions
Baselines: MemGPT MemoryBank Claude 3 Opus GPT-4 turbo GPT-4o Gemini 1.5 Pro Llama 3 70B Mixtral 8x22B
Builds on: Needle in a Haystack MemGPT MemoryBank ChapterBreak ToMi
Closest papers by meaning
- AMemGym: Interactive Memory Benchmarking for Assistants in Long-Horizon Conversations (ICLR 2026 · similarity 0.90)
- AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications (ICML 2026 · similarity 0.90)
- LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (ICLR 2025 · similarity 0.90)
- LOCA-bench: Benchmarking Language Agents Under Controllable and Extreme Context Growth (ICML 2026 · similarity 0.90)
- Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions (ICLR 2026 · similarity 0.89)
- Exploring Conversational Adaptability: Assessing the Proficiency of Large Language Models in Dynamic Alignment with Updated User Intent (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.