Research map / Architectures and efficiency / Attention and transformers
Kinaema: a recurrent sequence model for memory and pose in motion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Kinaema, a recurrent transformer-based sequence model with distributed GRU-gated memory that is trained for relative pose estimation to memory (Mem-RPE) and plugged into a DEBiT navigation agent for Mem-Nav, where agents navigate to goals seen during an unseen-goal priming sequence.
Problem
Enabling embodied agents in continuous operation to exploit observations gathered before an episode starts, i.e., to localize and navigate to goal images depicting previously seen space, while keeping memory update and query cost independent of history length.
Approach
A recurrent memory of N embeddings updated by a transformer (self-attention over memory after linear correction by observation features) and gated by a weight-shared GRU cell, supervised with relative pose estimation between a query image and the memory plus masked image modeling; the read-out is fed to a pose decoder and to an RL navigation agent.
Key results
- Kinaema achieves 21/41/63% on Mem-RPE at sequence length 200 and 10/21/37% at length 800, outperforming GRU (12/32/56 and 4/14/31), xLSTM, EMA and MooG.
- On Mem-Nav (HM3D val), the Kinaema-augmented DEBiT agent reaches 50% SPL on easy seen episodes vs 45% for DEBiT finetune, and 54% (+10) on hard seen episodes.
- Increasing memory embeddings improves Mem-RPE up to about N=20 embeddings; fewer, larger embeddings work better for fixed memory size.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Kinaema instead of the strongest recurrent baseline | Mem-RPE · rpetest, T=200; pose accuracy (<1m, 10°) | pose accuracy: 12 -> 21 | Helped |
| Use Kinaema instead of the strongest recurrent baseline | Mem-RPE · rpetest, T=800; pose accuracy (<1m, 10°) | pose accuracy: 5 -> 10 | Helped |
| Add priming-sequence Kinaema memory to DEBiT | Mem-Nav · navtest, hard seen | SPL (%): 44 -> 54 | Helped |
| Add priming-sequence Kinaema memory to DEBiT | Mem-Nav · navtest, easy unseen | SPL (%): 47 -> 46 | No clear effect |
| Add the transformer update block (ablation) | Kinaema · Mem-RPE rpeval, T=800; pose accuracy (<1m, 10°) | pose accuracy: 3 -> 13 | Helped |
| Add the gating block (ablation) | Kinaema · Mem-RPE rpeval, T=800; pose accuracy (<1m, 10°) | pose accuracy: 4 -> 13 | Helped |
| Randomize training sequence length (ablation) | Kinaema · Mem-RPE rpeval, T=800; pose accuracy (<1m, 10°) | pose accuracy: 4 -> 13 | Helped |
| Add masked image modeling loss (ablation) | Kinaema · Mem-RPE rpeval, T=800; pose accuracy (<1m, 10°) | pose accuracy: 4 -> 13 | Helped |
| Increase memory embeddings from 1 to 20 (ablation) | Kinaema · Mem-RPE rpeval, T=200; pose accuracy (<1m, 10°) | pose accuracy: 7 -> 24 | Helped |
| Increase memory embeddings from 20 to 50 (ablation) | Kinaema · Mem-RPE rpeval, T=800; pose accuracy (<1m, 10°) | pose accuracy: 13 -> 1 | Hurt |
| Use fewer, larger memory embeddings at fixed memory size (ablation) | Kinaema · Mem-RPE rpeval, T=200; pose accuracy (<1m, 10°) | pose accuracy: 4 -> 24 | Helped |
| Make the EMA decay factor trainable (ablation) | EMA · Mem-RPE rpeval, T=200; pose accuracy (<1m, 10°) | pose accuracy: 4 -> 6 | Helped |
Surprising findings
- Simple EMA memory with 153.6k entries is outperformed by a much smaller recurrent memory, showing memory size alone does not compensate for simple dynamics.
- Kinaema training seeds showed bimodal performance: either excellent or mediocre.
Limitations
- Model trained only for relative pose estimation and does not predict whether a goal was seen before.
- Training limited to sequences of length T=100, restricting performance on longer sequences.
- Single-embedding visual input per frame instead of patch-wise representation may limit future improvements.
Experimental setup
Datasets: HM3D Gibson
Benchmarks: Mem-RPE (HM3D val, sequence lengths 200 and 800) Mem-Nav (HM3D val)
Metrics: accuracy at <1m/10deg, <1m/90deg, <2m/90deg success rate SPL
Baselines: GRU EMA xLSTM MooG Truncated history DEBiT (zeroshot) DEBiT (finetune) h0-injection
Builds on: DEBiT DINOv2 PPO MooG xLSTM
Closest papers by meaning
- Learning with a Mole: Transferable latent spatial representations for navigation without reconstruction (ICLR 2024 · similarity 0.86)
- MemoNav: Working Memory Model for Visual Navigation (CVPR 2024 · similarity 0.85)
- Reasoning in Visual Navigation of End-to-end Trained Agents: A Dynamical Systems Approach (CVPR 2025 · similarity 0.85)
- Change-Robust Online Topological Memory for Long-Term Relocalization and Semantic Navigation (NeurIPS 2026 · similarity 0.85)
- SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation (ICML 2025 · similarity 0.85)
- 3D-Mem: 3D Scene Memory for Embodied Exploration and Reasoning (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.