SoundSpaces is a realistic acoustic simulation platform for audio-visual embodied AI research. From audio-visual navigation, audio-visual exploration to echolocation and audio-visual floor plan reconstruction, this platform expands embodied vision research to a broader scope of topics.

Click on the gif to view the video. Listen with headphones to hear the spatial sound properly!

Motivation

Moving around in the world is naturally a multisensory experience, but today's embodied agents are deaf---restricted to solely their visual perception of the environment. We introduce audio-visual navigation for complex, acoustically and visually realistic 3D environments. We further build SoundSpaces: a first-of-its-kind dataset of audio renderings based on geometrical acoustic simulations for two sets of publicly available 3D environments (Matterport3D and Replica), and we instrument Habitat to support the new sensor, making it possible to insert arbitrary sound sources in an array of real-world scanned environments.

Citing SoundSpaces

If you use the SoundSpaces platform in your research, please cite the following paper:

@inproceedings{chen20soundspaces,
  title     =     {SoundSpaces: Audio-Visual Navigaton in 3D Environments},
  author    =     {Changan Chen and Unnat Jain and Carl Schissler and Sebastia Vicenc Amengual Gari and Ziad Al-Halah and Vamsi Krishna Ithapu and Philip Robinson and Kristen Grauman},
  booktitle =     {ECCV},
  year      =     {2020}
}

@article{chen22soundspaces2,
  title     =     {SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning},
  author    =     {Changan Chen and Carl Schissler and Sanchit Garg and Philip Kobernik and Alexander Clegg and Paul Calamia and Dhruv Batra and Philip W Robinson and Kristen Grauman},
  journal   =     {arXiv},
  year      =     {2022}
}

Installation

1. Install habitat-lab v0.2.1 and habitat-sim from this commit
2. Install this repo into pip by running the following command:

pip install -e .

3. To use SoundSpaces 1.0, follow instructions on the dataset page to download the rendered audio data and datasets.
4. [New] To use SoundSpaces 2.0, add --audio flag while building Habitat-Sim from the source.

Usage

This repo renders audio-visual observations with high acoustic and spatial correspondence. It supports various visual-acoustic learning tasks, including audio-visual embodied navigation, acoustics prediction from egocentric observations, etc. In this repo, we provide code for training and evaluating audio-visual navigation agents. For other downstream tasks, please check out each paper's respective repo, e.g., visual acoustic matching and audio-visual dereverberation

Below we show some example commands for training and evaluating AudioGoal with depth sensor on Replica.

1. Training

python ss_baselines/av_nav/run.py --exp-config ss_baselines/av_nav/config/audionav/replica/train_telephone/audiogoal_depth.yaml --model-dir data/models/replica/audiogoal_depth

2. Validation (evaluate each checkpoint and generate a validation curve)

python ss_baselines/av_nav/run.py --run-type eval --exp-config ss_baselines/av_nav/config/audionav/replica/val_telephone/audiogoal_depth.yaml --model-dir data/models/replica/audiogoal_depth

3. Test the best validation checkpoint based on validation curve

python ss_baselines/av_nav/run.py --run-type eval --exp-config ss_baselines/av_nav/config/audionav/replica/test_telephone/audiogoal_depth.yaml --model-dir data/models/replica/audiogoal_depth EVAL_CKPT_PATH_DIR data/models/replica/audiogoal_depth/data/ckpt.XXX.pth

4. Generate demo video with audio

python ss_baselines/av_nav/run.py --run-type eval --exp-config ss_baselines/av_nav/config/audionav/replica/test_telephone/audiogoal_depth.yaml --model-dir data/models/replica/audiogoal_depth EVAL_CKPT_PATH_DIR data/models/replica/audiogoal_depth/data/ckpt.220.pth VIDEO_OPTION [\"disk\"] TASK_CONFIG.SIMULATOR.USE_RENDERED_OBSERVATIONS False TASK_CONFIG.TASK.SENSORS [\"POINTGOAL_WITH_GPS_COMPASS_SENSOR\",\"SPECTROGRAM_SENSOR\",\"AUDIOGOAL_SENSOR\"] SENSORS [\"RGB_SENSOR\",\"DEPTH_SENSOR\"] EXTRA_RGB True TASK_CONFIG.SIMULATOR.CONTINUOUS_VIEW_CHANGE True DISPLAY_RESOLUTION 512 TEST_EPISODE_COUNT 1

5. Interactive demo

python scripts/interactive_demo.py

5. [New] Training continuous navigation agent

python ss_baselines/av_nav/run.py --exp-config ss_baselines/av_nav/config/audionav/mp3d/train_telephone/audiogoal_depth_ddppo.yaml --model-dir data/models/ss2/mp3d/dav_nav CONTINUOUS True

SoundSpaces 1.0

We provide acoustically realistic audio renderings for Replica and Matterport3D datasets. The audio renderings exist in the form of pre-rendered room impulse responses (RIR), which allows users to convolve with any source sounds they wish during training. See dataset for more details.
Note that we do not open source the rendering code at this time.

SoundSpaces 2.0

SoundSpaces 2.0 is a fast, continuous, configurable and generalizable audio-visual simulation platform that allows users to render sounds for arbitrary spaces and environments. As a result of rendering accuracy improvements, the rendered IRs are different from SoundSpaces 1.0.

Some common issues

• To render the monaural audio, set the microphone type to Ambisonics and channel size to 1 for now.
• If you run into invalid pointer issues, import quaternion before habitat_sim as a workaround.
• See this issue for solutions to GLIBC version issues

Contributing

See the CONTRIBUTING file for how to help out.

License

SoundSpaces is CC-BY-4.0 licensed, as found in the LICENSE file.

The trained models and the task datasets are considered data derived from the correspondent scene datasets.

• Matterport3D based task datasets and trained models are distributed with Matterport3D Terms of Use and under CC BY-NC-SA 3.0 US license.
• Replica based task datasets, the code for generating such datasets, and trained models are under Replica license.