Research map / Vision-language and multimodal / Cross-modal learning
See, Hear, Explore: Curiosity via Audio-Visual Association
Victoria Dean, Shubham Tulsiani, Abhinav Gupta
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use audio-visual association novelty as intrinsic reward instead of visual future-prediction error | exploration policy on Atari · Atari (12 games) | games outperforming future prediction curiosity: 8 of 12 | Improved |
| use audio-visual association novelty as intrinsic reward instead of disagreement | exploration policy on Atari · Atari (12 games) | games outperforming disagreement: 9 of 12 | Improved |
| use audio-visual association novelty for exploration | exploration policy in Habitat, Apartment 0 · Habitat, Apartment 0 | time to full state coverage: 3 times faster | Improved |
| use audio-visual association novelty for exploration | exploration policy in Habitat, Apartment 0 · Habitat, Apartment 0 | visits to rare states: about 8 times more frequently | Improved |
| combine SHE and future-prediction intrinsic rewards (ablation) | Atari exploration policy · Atari (12 games) | games outperforming future prediction curiosity: 10 of 12 | Improved |
| combine SHE and future-prediction intrinsic rewards (ablation) | Atari exploration policy · Alien | average final reward: 359 -> 593 | Improved |
| add audio-visual inputs to the future-prediction baseline (ablation) | future prediction curiosity · Atari | extrinsic reward | No clear effect |
| add audio-visual inputs to the RND baseline (ablation) | RND · Gravitar | extrinsic reward | No clear effect |
| use audio-visual association rather than future prediction (ablation) | curiosity-driven exploration under noisy audio and visual inputs · Asterix, SpaceInvaders, and MsPacman | extrinsic reward under noise | Mixed |
| use audio-visual association curiosity in sticky-action Atari environments (ablation) | SHE · Atari sticky-action environments | exploration performance | No clear effect |
Closest papers by meaning
- Learning to Set Waypoints for Audio-Visual Navigation (ICLR 2021 · similarity 0.89)
- SoundSpaces: Audio-Visual Navigation in 3D Environments (ECCV 2020 · similarity 0.88)
- RILA: Reflective and Imaginative Language Agent for Zero-Shot Semantic Audio-Visual Navigation (CVPR 2024 · similarity 0.87)
- Curiosity-driven Exploration by Self-supervised Prediction (ICML 2017 · similarity 0.87)
- CAVEN: An Embodied Conversational Agent for Efficient Audio-Visual Navigation in Noisy Environments (AAAI 2024 · similarity 0.86)
- Wonder Wins Ways: Curiosity-Driven Exploration through Multi-Agent Contextual Calibration (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
