Vulkan, Metal and...?

[kblaschke]

As Mischa already posted in his wishlist, Vulkan would probably be the 3D API of choice for most platforms. There are currently two platforms where Vulkan cannot be used (at loeast not directly):

• macOS/iOS/tvOS: Apple does not support Vulkan. Instead, Apple left the Khronos Vulkan working group at some point, went with the early draft at this time and created their own API, Metal. There's MoltenVK, which is a translation layer from Vulkan to Metal. The only drawback here is that Metal only supports a subset of Vulkan's current feeatures, so the visualizer would have to be limited to those.
• Browsers: Currently, the only 3D API supported within browsers is WebGL, which is similar in funcitonality to OpenGL ES. There's a working draft for a new "WebGPU" API which would run on top of Vulkan, Metal or DirectX 12, and thus be relatively similar to Vulkan. It's still an early draft and only experimentally supported in Chrome browsers as of the time of writing. Plus. the shading language used is hugely different in syntax from GLSL/HLSL, but it seems the standard will also support SPIR-V shaders, which can be compiled from GLSL.

Then there might be platforms like (older) mobile devices which don't have Vulkan support. It's definitely worth discussing whether we'd need a more flexible renderer with different backends (e.g. both Vulkan and OpenGL), which would improve compatibility, but require more implementation work as the rendering API would have to be abstracted (which isn't generally a bad thing to do anyway). In this case, writing a native Metal renderer would also be possible. Since different rendering APIs have different features, presets would have to be aware of that - which also applies to shader and GPU features/specs as well.

In the end, the core library would be more like a specialized 3D engine for audio visuals, so we may treat it as such.

[tukkek]

Vulkan was never meant to be used directly but instead to create intermediate graphics-libraries and engines with. See this post by a senior moderator for the Khronos Group (responsible for Vulkan, Open GL, Web GL and more):

Vulkan is a very low-level API and while it will probably be useful in that regard, not everyone really needs that low-level nature. So Vulkan makes for a good API to build other “higher-level-but-still-not-scene-graph-level” graphics APIs out of. https://community.khronos.org/t/layering-apis-on-top-of-vulkan/4978

I'm not sure what the Vulkan eco-system is like in 2024 but I would suggest against using Vulkan directly unless bare-metal access is strictly necessary. As visualizers were running on decades-old non-accelerated hard-ware, I don't see why this would be a necessity for Project M.

[kblaschke]

The visualizer is actually an engine, and the presets are using the engine to draw stuff, so it's definitely within the mentioned use case.

On the other hand, if you say Vulkan should be a no-go, then what alternatives would you suggest that fit into the requirements, e.g. being available on most platforms etc?
Besides OpenGL, I actually don't know any.

What would be an option is using a third-party 3D engine and let this handle all the low-level stuff. Ogre3D for example. This could also seriously decrease the amount of code needed to be written, but possibly add some more overhead.

Same as with the Python thing in the other thread, always remember a visualizer is just a small plug-in loaded into a probably already large software. The shared library should not be tens of megabytes in size and use up a GB of RAM before even loading a preset.

[kblaschke]

And to add to this, Vulkan might be low-level, but it's not as bad as most people think. Setting up stuff needs more code because the API is very explicit, you have to take care about a few more things than with OpenGL, but that's about it. The actual rendering is basically the same once you wrapped a few things into classes.

[tukkek]

My suggestion would've been what you said: finding an engine; frame-work; or higher-level graphics library rather than doing things as low-level as possible outside of a driver. I can't even draw a shader myself but I wanted to comment because I have seen the Vulkan creators say that the technology wasn't built to be used directly but more to allow something like Open GL to be built on-top of it.

This page lists many higher-level Vulkan-based libraries and engines, including SDL https://github.com/vinjn/awesome-vulkan?tab=readme-ov-file#libraries

I am certainly not advocating for Project M to be built with Unreal Engine; or to do like Blizzard did and bundle a 900-mega-bytes Electron build just to draw the main menu for a game (seriously)... However to say 4 mega-bytes, or 10 or even 100 in 2024 is a no-go, seems to me like throwing the baby away with the bath-water.

[kblaschke]

Yeah, some middleware to reduce complexity may make some sense.

One thing to keep in mind though is that the visualizer is being embedded into an existing application and thus will have to use the application-provided rendering context (a Vulkan instance, or an OpenGL context like projectM needs) instead of creating its own full rendering stack, which will require adding OS-specific windowing APIs and other stuff that may even conflict with the application embedding the library. Plus, passing the rendered image back to the application should ideally happen within the GPU to avoid the overhead of downloading several megabytes of image data per frame to RAM and back into the GPU to display it - this also only works if both the application and the visualizer share the same rendering API instance.

Using some existing engine often means that the engine will take control of all those steps - creating a window, the renderer instance, and managing all resources. If choosing such a middleware, this needs ot be taken into account, plus portability.

Regarding portability, using Vulkan alone is probably not even enough if we want to support WASM, as stated above, so there may even be different rendering backends for each platform the library is built for.

In the end, the preset authors don't have to care about all this, they should just specify geometry, textures and shaders - or just combine existing effects - to create the visuals. The library AKA the engine then does all the low-level work of getting the scene to the GPU for rendering.

[tukkek]

I'm aware this is a very minor thing but if a solution exists that has built-in support for skins, that could be a fun throw-back feature too!