Research map / Large language models / AI and society
Working Memory Capacity of ChatGPT: An Empirical Study
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Assesses ChatGPT's working memory capacity via verbal and spatial n-back tasks with several variants, finding a capacity limit near n=3 similar to humans and proposing n-back as an LLM working-memory benchmark.
Problem
Whether LLMs such as ChatGPT have a working memory capacity limit comparable to humans, and whether n-back performance can serve as a benchmark for LLM working memory.
Approach
Applies human cognitive-science n-back paradigms (verbal and spatial, with noise, feedback, chain-of-thought, abstract-reasoning and grid-size variants) to prompted LLMs, using d' with a threshold of 1 to define capacity limit, and compares across several LLMs.
Key results
- Verbal base-version ChatGPT d' drops to around 1 at n=3, indicating a capacity limit of about 3 similar to humans.
- Adding noise to verbal inputs significantly reduced capacity (noise variant not significant from 1 even at n=1, Kruskal-Wallis p=0.14).
- GPT-4 far exceeds other tested LLMs on the verbal base task, while open-source models (Bloomz, ChatGLM, Vicuna) show very low capacity and are nearly indistinguishable.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase n from 1 to 3 in the base verbal task | ChatGPT (GPT-3.5) · Verbal n-back | d' | Hurt |
| Add noise symbols to verbal stimuli (ablation) | ChatGPT (GPT-3.5) · Verbal n-back | working memory capacity | Hurt |
| Add step-by-step reasoning prompt (ablation) | ChatGPT (GPT-3.5) · Spatial n-back | performance | Helped |
| Use spatial rather than verbal stimuli (ablation) | ChatGPT (GPT-3.5) · n-back task | working memory capacity | Hurt |
| Require abstract spatial reasoning (ablation) | ChatGPT (GPT-3.5) · Spatial n-back | working memory capacity | Hurt |
| Increase spatial grid size (ablation) | ChatGPT (GPT-3.5) · Spatial n-back | working memory capacity | Helped |
| Use GPT-4 instead of the open-source LLMs tested | Verbal n-back · Base verbal task | working memory capacity | Helped |
| Use the tested open-source LLMs instead of GPT-3.5/GPT-4 | Verbal n-back · Base verbal task | working memory capacity | Hurt |
Surprising findings
- Spatial and verbal noise variants show near-chance performance, with noise severely reducing working memory capacity even at n=1 for verbal-with-noise.
- Larger grid sizes gave higher working memory capacity for the LLM, contrary to the expectation of greater difficulty.
Limitations
- GPT-4 was not tested beyond n=3 due to API cost.
- Spatial tasks and abstract reasoning variants lack human comparison data.
- Only one model family (ChatGPT) analyzed in depth; results limited to gpt-3.5-turbo at temperature 1.
Experimental setup
Benchmarks: verbal n-back spatial n-back
Metrics: d' (detection sensitivity) hit rate false alarm rate accuracy
Baselines: GPT-4 Bloomz-7B Bloomz-7B1-mt ChatGLM-6B_v1.0 ChatGLM-6B_v1.1 Vicuna-7B Vicuna-13B
Builds on: chain-of-thought prompting executive attention hypothesis
Closest papers by meaning
- LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (ICLR 2025 · similarity 0.86)
- MemoryBank: Enhancing Large Language Models with Long-Term Memory (AAAI 2024 · similarity 0.86)
- Simulating Human Memory with Language Models (NeurIPS 2026 · similarity 0.86)
- Longer Context, Deeper Thinking: Uncovering the Role of Long-Context Ability in Reasoning (NeurIPS 2025 · similarity 0.85)
- Large Language Models Think Too Fast To Explore Effectively (NeurIPS 2025 · similarity 0.84)
- Simple linear attention language models balance the recall-throughput tradeoff (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.