Research map / Video / Video segmentation and tracking
PPMStereo: Pick-and-Play Memory Construction for Consistent Dynamic Stereo Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PPMStereo, a memory-buffer dynamic stereo matcher that picks the top-K most reliable reference frames (via confidence and redundancy-aware relevance scores) and weights them for attention-based aggregation, built on DynamicStereo.
Problem
Temporally consistent depth estimation from stereo video requires long-range temporal modeling, but sliding-window attention treats all frames equally and full-frame memory is computationally prohibitive.
Approach
A Pick-and-Play Memory: a Quality Assessment Module scores each stored reference frame by confidence plus redundancy-aware similarity to select top-K frames into a compact buffer, then dynamic weighting and temporal positional encoding modulate those entries before attention-based read-out inside an iterative GRU refiner.
Key results
- On Sintel clean/final with SF+DR training, PPMStereo reaches TEPE 0.62 / 1.11 and 3-px error 5.19% / 7.64%, versus BiDAStereo's 0.75 / 1.22 TEPE.
- On Dynamic Replica (first 150 frames) PPMStereo reaches TEPE 0.057 when trained on DR+SF, below BiDAStereo's 0.062.
- Ablation on Sintel final: the pick and play components each improve over the baseline (TEPE 1.37 to 1.14 and 1.17), and combining them gives 1.11.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the feature extractor with Video Depth Anything (ViT-Small) | PPMStereo · Sintel Final | TEPE: 1.11 -> 1.02 | Helped |
| Add PPM memory construction instead of the baseline without memory modules (ablation) | PPMStereo · Sintel Final | TEPE: 1.37 -> 1.11 | Helped |
| Add PPM memory construction instead of the baseline without memory modules (ablation) | PPMStereo · Dynamic Replica | TEPE: 0.074 -> 0.057 | Helped |
| Increase memory length from K=1 to K=5 (ablation) | PPMStereo · Sintel Final | TEPE: 1.18 -> 1.11 | Helped |
| Increase memory length from K=5 to K=7 (ablation) | PPMStereo · Sintel Final | TEPE: 1.11 -> 1.10 | Helped |
| Replace PPM frame selection with random selection (ablation) | PPMStereo · Sintel Final | TEPE: 1.11 -> 1.26 | Hurt |
| Replace PPM frame selection with RMem (ablation) | PPMStereo · Sintel Final | TEPE: 1.11 -> 1.16 | Hurt |
| Add the pick process (ablation) | PPMStereo baseline · Sintel Final | TEPE: 1.37 -> 1.14 | Helped |
| Add the play process (ablation) | PPMStereo baseline · Sintel Final | TEPE: 1.37 -> 1.17 | Helped |
| Add both pick and play processes (ablation) | PPMStereo baseline · Dynamic Replica | TEPE: 0.074 -> 0.057 | Helped |
| Add confidence scoring to the pick process (ablation) | PPMStereo pick process · Sintel Final | TEPE: 1.17 -> 1.14 | Helped |
| Add similarity scoring to confidence scoring (ablation) | PPMStereo pick process · Dynamic Replica | TEPE: 0.058 -> 0.057 | Helped |
Surprising findings
- Random frame selection underperforms single-neighbor memory (MemFlow) on Sintel final, while on Dynamic Replica the random policy performs comparably to advanced variants.
- Naively using all 20 frames as memory causes out-of-memory errors during training.
Limitations
- Cannot proactively distinguish dynamic from static regions, which matters for temporal consistency.
- Occasional inconsistencies in textureless areas (e.g., blank walls) and transparent surfaces (e.g., glass).
Experimental setup
Datasets: SceneFlow Dynamic Replica
Benchmarks: Sintel Stereo (clean and final) Dynamic Replica South Kensington SV
Metrics: delta_3px TEPE delta_1px^t delta_3px^t
Baselines: CODD RAFT-Stereo DynamicStereo BiDAStereo CREStereo MemFlow XMem RMem
Builds on: DynamicStereo RAFT-Stereo MemFlow XMem Video Depth Anything
Closest papers by meaning
- Stereo Anywhere: Robust Zero-Shot Deep Stereo Matching Even Where Either Stereo or Mono Fail (CVPR 2025 · similarity 0.87)
- Mono2Stereo: A Benchmark and Empirical Study for Stereo Conversion (CVPR 2025 · similarity 0.87)
- Stereo Risk: A Continuous Modeling Approach to Stereo Matching (ICML 2024 · similarity 0.87)
- RobIA: Robust Instance-aware Continual Test-time Adaptation for Deep Stereo (NeurIPS 2025 · similarity 0.87)
- FoundationStereo: Zero-Shot Stereo Matching (CVPR 2025 · similarity 0.86)
- Stereo4D: Learning How Things Move in 3D from Internet Stereo Videos (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.