Add audio engine v2 #15839

docEdub · 2024-11-18T07:09:19Z

This change adds the initial audio engine v2 implementation and introduces the API for sound and streaming sound creation and playback, in addition to audio buses (formerly known as SoundTracks).

packages/dev/core/src/Audio/v2/audioPositioner.ts

packages/dev/core/src/Audio/v2/spatialAudioListener.ts

packages/dev/core/src/Audio/v2/spatialAudioTransform.ts

ryantrem · 2024-11-18T17:46:45Z

Am I remembering correctly from previous conversations that the new audio engine will implement the existing interfaces, or no?

bjsplat · 2024-11-18T17:56:38Z

Snapshot stored with reference name:
refs/pull/15839/merge

Test environment:
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15839/merge/index.html

To test a playground add it to the URL, for example:

https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15839/merge/index.html#WGZLGJ#4600

Links to test babylon tools with this snapshot:

https://playground.babylonjs.com/?snapshot=refs/pull/15839/merge
https://sandbox.babylonjs.com/?snapshot=refs/pull/15839/merge
https://gui.babylonjs.com/?snapshot=refs/pull/15839/merge
https://nme.babylonjs.com/?snapshot=refs/pull/15839/merge

To test the snapshot in the playground with a playground ID add it after the snapshot query string:

https://playground.babylonjs.com/?snapshot=refs/pull/15839/merge#BCU1XR#0

docEdub · 2024-11-18T18:40:23Z

Am I remembering correctly from previous conversations that the new audio engine will implement the existing interfaces, or no?

Yes, this is something we discussed as a possibility.

It is not needed. The sound plays without adding it to the document body.

This is particularly important for Vision Pro because it suspends/interrupts the audio context when exiting immersive mode.

The WebAudio-based streaming sound class's underlying `HTMLMediaElement` play and stop functions don't have `waitTime` parameters like the static sound class's `AudioBufferSourceNode` does. I tried implementing them in the streaming sound class using `setTimeout`, but the timing is too flaky, so this change removes the parameter completely. Note that the `waitTime` parameter for the static sound classes remains, but it is moved to the end of the parameter list so the streaming and static sound class play signatures are consistent with each other.

bjsplat · 2025-03-04T17:42:27Z

Visualization tests for WebGPU

https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15839/merge/testResults/webgpuplaywright/index.html

bjsplat · 2025-03-04T17:44:50Z

WebGL2 visualization test reporter:

https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15839/merge/testResults/webgl2playwright/index.html

RaananW

🎉

RaananW reviewed Nov 18, 2024

View reviewed changes

packages/dev/core/src/Audio/v2/audioPositioner.ts Outdated Show resolved Hide resolved

packages/dev/core/src/Audio/v2/spatialAudioListener.ts Outdated Show resolved Hide resolved

packages/dev/core/src/Audio/v2/spatialAudioTransform.ts Outdated Show resolved Hide resolved

docEdub added the new feature label Nov 18, 2024

docEdub force-pushed the audio-engine-v2 branch from c558cd4 to a8e734a Compare November 22, 2024 00:00

docEdub added 24 commits November 21, 2024 19:22

Add underscore to internal classes to hide them from BABYLON namespace

cb88f16

Add audio engine unlock()

1aee57c

Add preload instance API to streaming sound classes

1082bd4

Remove WebAudioEngine.isUnlockedPromise

80c1ee3

Don't return sound instances from sound play functions

ffeb598

Add underscore prefix to internal sound instance classes

1d0f02c

Put streaming sound options in alphabetical order and expand string type

abb88f7

Fix static sound file extension regex

e4fc35c

Add audio engine volume property

57da08a

Disable no-internal-modules check in Audio/index.ts

248f298

Fix static sound file extension regex for URLs with no query parameters

eaf7362

Remove unnecessary bind() calls

3e6ad6c

Fix play/pause/resume on static and streaming sounds

dc42780

Fix streaming sound state issues when audio context is not running

19b01a6

Don't add streaming sound media element to document body

e41c85d

It is not needed. The sound plays without adding it to the document body.

Remove completed todo

1e5b4e1

Get sound URLs containing a # working

c95b4ac

Add audio engine resumeOnPause options

c957e15

Handle streaming sound play failures when another user gesture is needed

88a2e5d

This is particularly important for Vision Pro because it suspends/interrupts the audio context when exiting immersive mode.

Fix doc comment

54ec2f1

Make AudioEngineV2.audioContext a variable and drop offline context

cb21e9d

Preload streaming sound instance before handling autoplay option

b262503

Implement sound currentTime getter

d52cc36