Some considerations on an audio plugin

[alnitak]

Perhaps now I can argue with a bit more knowledge than before, so I'd like to share my findings, although I'm sure I lack a knowledge base about WASM, JS, GC, and so on.

I have read many discussions about issues encountered in implementing audio callbacks in Dart, passing audio data to/from native and Dart, with the possibility that GC intervention may interfere with the real-time nature of an application such as an audio synthesizer.

My suggestion is to avoid using Dart to manage audio data and/or audio callbacks, neither within an isolate. Here are the problems:

• It's somewhat complicated at the moment to manage communications between C and Dart back and forth from different threads.
• Even if we implement SendPort / ReceivePort between threads, we'll encounter delays caused by copying send/receive messages, which is unacceptable for a real-time audio app.
• The native heap memory of native C cannot be shared between C thread and Dart.

Although things may change, in my opinion, making these functionalities available will break Dart SDK philosophy, i.e., safety and stability.

AFAIK, the garbage collector does not influence native FFI memory. So, my approach would be to use Flutter merely as a trampoline to send messages to native C audio. These messages can be managed in an audio loop (similar to a game loop or a render loop for OpenGL).

In flutter_soloud, there isn't a native audio loop (yet?). Many functions are called directly (getPosition, seek, setVolume, etc.), while others are called through an isolate because of their asynchronous nature (loadFile) or because they should manage active sounds through the audio loop. The audio loop has been implemented into this isolate to avoid the hassle of implementing it natively.

One thing that may change is the support for async callbacks discussed in this issue.
Another useful proposal that could help is the Shared Memory Multithreading proposal and discussed here.

Regarding the WASM part, I didn't understand the GC behavior well. I have compiled the C sources into a .wasm and a .js. I don't think the Dart VM will use its GC to manage the compiled WASM module, I think it can manage the WASM compiled with the --wasm of the app and not a second linked module. Anyway, I can now successfully call WASM functions with dart:js_interop using the simplest way with JS() export functionName() even when the native function uses pointers. That was the only need I had.
Also having a standalone compiled WASM module, I don't think that compiling the web app with WASM will work since there may be problems having 2 modules to communicate to each other. Have to try.

I hope these points are helpful. These are my thoughts, but they are certainly limited, and there may be other considerations to take into account. I hope I haven't stated the obvious.

[filiph]

FWIW, I concur with trying to leave all work with audio data itself to the C side of things. That's what we're currently doing, right?

As for the other things, I simply don't know enough to have a worthy opinion. I'll reach out to Maksim who is about 100x more knowledgeable about these things than I am.

[alnitak]

Yes, all the "dirty work" is done on the C side. Things may be different using that "C side" compiled to WASM module for the web.