Scripting language

[kblaschke]

Milkdrop (and AVS) used their very own, simple scripting language to enable preset authors to control many parts of the visuals. While this works quite well, the expression language doesn't support a lot of features like functions and user-defined data structures. There's also no way of controlling the whole rendering process within these expressions - they're solely responsible to control some variables for the predefined effects.

The new visualizer should ideally provide a more flexible way for authors to control the rendering process. While there will be some contraints, the preset should be able to control the following aspects:

• Setting up the scene composition. This includes rendering stages, geometry, camera type, FOV and movement and other things.
• Define individual rendering steps including bloom, blur and post-processing effects.
• Create, load and control any geometry to be rendered. Simple quads/triangles, particles or complex textured meshes.
• Render more complex pipelines, e.g. render two separate scenes and blend them together in a third step.
• Process audio data, certain user input (e.g. touch events) and use these to control the rendering.
• Procedurally generate meshes, textures and even shaders.
• Write and export functions to be reused within the preset (or other presets wihtin an effect library)

There should be different entry points into the scripts, both one-off calls (e.g. when loading the preset) and also events the preset can react on. The main function would be the per-frame render/update function which does the majority of the work to setup the renderer for a single frame.

Language

There are many scripting languages available which can be embedded. The most popular ones are:

Lua

A lightweight, relatively simple and widely adopted scripting language used in possibly thousands of software projects, many of those being games.

Lua has a very basic type systems, e.g. it has no strongly-typed variables. Lua also doesn't support complex data structures, only simple variables containing a single value or tables, which are basically hash tables with key/value pairs.

Functions can be freely defined and used within a Lua scope, and the application can also provide functions which are routed back into the C/C++/Rust/etc. code of the underlying application to let the user control things and request data for example.

Lua has an optional standard library of functions which can easily be disabled for safety purposes, e.g. the os library which would allow users to execute arbitrary code.

Python

A widely used language, which can be used as a scripting interpreter within other applications or as a standalone language to write any kind of application in. It ships with a (bytecode) compiler to improve loading times and perform some optimizations.

Python is is more like a full programming language, which also supports modules, some object-oriented features and more. This also means that a Python interpreter is relatively large in comparison to Lua, and also needs a number of basic modules to be loaded and available to properly work.

Since Python is relatively big and rich in features, it's relatively hard to disable certain parts to make it safe to use with untrusted code. There might be some sandboxing feature available though.

Conclusion

IMO, Lua would be the best option here as it provides enough flexibility, is safe, fast and really small (<200 KB) and embeds very easily. There's also a huge userbase knowing how to write code in that language as well, as most games with modding support use Lua for that purpose. LuaJit could also be used to further improve runtime performance if needed.

While Python will surely offer more features, it'll increase the library size by several megabytes, isn't easy to secure and prevent malicious code from being executed.

Other scripting hosts are also available, e.g. NodeJS (which has basically the same issues as Python).

If anyone else has suggestions for a language more similar to Lua, please post them!

[tukkek]

I have worked extensively with both Lua and Python as embedded languages (both developing scripts and adding script support to C++ engines). There isn't, in my opinion a single reason to use Lua over Python.

Python's syntax is world-class, its API is infinitely ahead of Lua and its popularity is unparalleled as the second most-wanted language only behind Java Script https://survey.stackoverflow.co/2023/#section-admired-and-desired-programming-scripting-and-markup-languages

Even the common myth that Lua is faster, often brought-up as Lua's one saving-grace in the exact debate of Lua versus Python is also a blatant lie as both have equivalent performance: https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/python3-lua.html

it's relatively hard to disable certain parts to make it safe to use with untrusted code

Python has an industry-standard security-auditing system that is an integral part of the core API, much like Java's security-managers https://peps.python.org/pep-0578/

a Python interpreter is relatively large in comparison to Lua,

Considering the data for Project M is almost 30 mega-bytes, I can's see Python's size, which is a small fraction of that, being a real issue (Debian's libpython3.9-minimal is 4 mega-bytes, uncompressed).

Obviously when discussing disk-space, memory and network band-width in 2024, anything below giga-bytes is irrelevant, let alone individual mega-bytes. If Project M was first-and-foremost created for embedded devices, then maybe I'd agree but not only that's not the case as far as I'm aware of but even then I'd still suggest something like Micro Python (200k of code and 10k of memory foot-print) over Lua.

Hopefully I have made it clear that using Lua over Python would be a significant detriment to this project, with no tangible benefit.

[kblaschke]

Object orientation shouldn't be needed for preset authoring. That one was just meant for the actual library, which will be vastly more complex than the presets themselves.

Python might be a good candidate, if the actual interpreter can be made secure in a way that any OS-related functionality could be stripped. Otherwise, this would have the same security implications as running a Windows DLL from a random Internet site on your machine.

[tukkek]

If object-orientation is useful to the point of being a deal-breaker for the engine, it could be just as useful for visualization-artists. Seems oddly premature to me to determine it's irrelevant a day into brain-storming ideas. I know I have used object-orientation in scripts where the developers hadn't expected me to do so (a project I'm actively working on right now, as matter-of-fact) and it has been hugely helpful.

If using Python, for example, artists could maintain and share utility-classes or entire libraries, which I don't believe is easy to do within Milk Drop (or at least not as elegantly but by copy-pasting and editing sections).

Imagine you wanted to add some Markov-generated-text to your visualization. It would certainly be easier to organize your script within a display and rendering class; and another for the Markov generator. Why not use a whole-sale pure-Python class as-is? https://github.com/jkratz55/markov-python/blob/master/MarkovLanguage.py

It seems to me that classes would be no less powerful and creative script-side than they would be server-side.

As for security, the auditing system (PEP 578) should be able to do just that, I believe, as I have linked to before.

[tukkek]

Here's a 10-lines solution for security management using Python's auditing system. Not as much a code contribution as just me wanting to get some hands-on experience out of curiosity with this topic:

import sys

BLOCK=['open']
ALLOW=['urllib','urllib.request']

def block(event,arguments):
  if event in BLOCK:
    raise Exception(f'Blocked: {event}.')
  if event=='import':
    module=arguments[0]
    if module not in ALLOW:
      raise Exception(f'Blocked import for "{module}".')

sys.addaudithook(block)

try:
  print('Create a new file.',file=open('test.txt','w'))
except Exception as e:
  print(e)

try:
  import urllib.request
  urllib.request.urlopen('https://google.com')
except Exception as e:
  print(e)

try:
  import glob
  print('User files:',glob.glob('**',recursive=True))
except Exception as e:
  print(e)

# Output:
'''
Blocked: open.
Blocked: open.
Blocked import for "glob".
'''

Note that just blocking open would immediately block every single case by itself (since urllib and glob use open() internally for obvious reasons) but I wanted to test a simple module white-list too with the last two cases.

For completeness' sake, let me just point out that the documentation states:

1. The equivalent C++ PySys_AddAuditHook() should be used as native hooks are called first for extra security.
2. Obviously, if one allows certain Pandora's Box modules to be imported, then no security can be guaranteed at all (such as the subprocess module, as methods like subprocess.run() will allow scripts to execute literally any available binary or shell command)...

[kblaschke]

Seems oddly premature to me to determine it's irrelevant a day into brain-storming ideas.

Didn't say it's irrelevant, just not as important. I definitely see the advantages of using Pythin, and I'm also not a huge fan of Lua as it is actually an oversimplified language.

The big question is, how much scripting ability and features do preset authors really need here? The scripting language should provide:

• Syntax features to write proper code - with functions, variables etc.
• Access to some specific library-provided functionality to access preset details and setting up the rendering process
• Utility functions and libraries to generate noise, perform vector math and other visualizer-related stuff

It should not provide or allow these features:

• Filesystem access (only access a list of library-provided files should suffice, e.g. preset textures and shaders).
• System calls like threading, memory and process management.
• Any type of network/socket access.
• Loading arbitrary Python modules.

Back in the days when Milkdrop and AVS were initially written, the internet was still young and malicious actors relatively rare, as most people weren't always connected and just a few applications actually required the internet to work. Nowadays, it's sadly a different story. If a software allows any kind of automated code execution, e.g. by loading a preset downloaded from some website/forum, then it's really big safety risk for the user. Python itself was created as a general-purpose programming language which one could use to do basically anything, including bad stuff. A large portion of today's password stealers are actually written in Python because it's so convenient and felxible. I personally don't want that negative press that the visualizer is the root source of a malware attack, which isn't too far-fetched if it makes its way into players like VLC and Kodi.

Maybe I'm taking this too serious because I was working in the IT security industry for the past years, but I'd say you can't ever take this issue lightly. The xzutils hack was a good example for this, if there's no real focus on code safety (and reviews, in this case).

About the size:

The thing with the visualizer is that it ideally should be self-contained in a single .DLL/.so/dylib file, without (many) external dependencies in regards to additional shared libraries, code files etc. to make it easier to distribute and use it, and also keep it small. So one will integrate a visualizer that takes 100 MB or more of disk space without any presets, plus needs a GB of RAM to load all the depesnencies. Plus, having such a huge ecosystem makes it more and more complicated to keep everything portable, especially when it comes to WASM or embedded/mobile platforms. projectM is a good example - it's just a single library, it has no external dependencies except OpenGL (and GLEW on Windows for loading the GL functions) and is less than 300 KB in size. While the new visualizer will surely get bigger and will have more dependencies, this should always be taken into account.

If there's a Python library that can be fully embedded into the library, with all the standard modules included - that'd be perfect.

[tukkek]

The big question is, how much scripting ability and features do preset authors really need here?

My only experience fiddling with visualizations was way back in the day but I can tell what makes me happy when I'm writing scripts and read an API I know will be easy to understand and use:

• Many pre-defined entry-point functions (such as: render() which is called every tick (or frame); load() or setup() which is called once; fadein() and fadeout() for transitions...
• Gives me global access to the functionalities I need, properly organized into objects (such as preset.detail or engine.fps or noise.generate().

(I don't mean anything in particular other than the overall structure in general).

This is a silly example of a script that adds more trees to a Mine Craft-like game. Of note is how it interacts with the engine by using the global minetest object and the engine interacts with it through the minetest.register call-backs.

https://github.com/mt-mods/moretrees/blob/master/init.lua

The reason I choose this as an example is because it handles a deceiving amount of complexity (especially with the other included scripts) while also being simple and elegant to understand, read and write, at least at surface-level.

Also the engine and scripting API itself is incredibly extensive and performance-sensitive like Project M as it allows for scripts to perform per-tick operations on servers with a large number of players. I also have some familiarity with it as I briefly contributed engine code.

It should not provide or allow these features

I believe all of these are achieved in the 10 lines of code I've sent previously. Letting users organize their code on different files (or share files between presets) would require allowing these files to be imported from the main script (like in the example above) but this can be added later.

Basically all of Python's input-output system is done through open() either explicitly or internally by the core library. We can also block all of the IO events explicitly, which are documented - or just block all events then remove the harmless ones as scripts need to use them.

Maybe I'm taking this too seriously

Not at all. I don't think it's realistic that visualizations will become an attack vector but regardless, end-user safety should indeed be taken seriously, especially when it's a simple problem to solve which also has already been fundamentally solved.

About the size

It's a trade-off. I agree needing a giga-byte of RAM is ludicrous but it's also possible to shoot yourself in the foot by being radical and not using the best tools available out there to work with.

A Python library that can be fully-embedded with all the standard modules included would be perfect.

I added Python scripting support to a pre-existing C++ engine and don't remember having a single problem building; deploying or running it when it came to dependencies or even having to install anything in the remote-server to get it working. It was all the way back in 2013 so take my account for what it's worth.

Python is only behind Java Script as a scripting language. It is ubiquitous as embedded scripting on projects big and small. It is the very definition of "industry-standard" and I don't imagine it's going to be more trouble than it's worth, considering the benefits it'd provide for visualization-artists.

[tukkek]

With scripting support, it would be interesting to allow visualizations to setup ahead of time, such as with a load event or load() method.

(Relatively) longer operations shouldn't be encouraged during visualization execution as even a less-of-a-second delay is noticeable when watching high-frame-rate play-back. However, no one would mind a 2 or 3 second delay as visualizations are first loaded as a whole, between opening and the start of rendering the first visualization.

This would open a large possibility-space for (relatively) processing-intensive visualizations, while ensuring smooth play-back isn't affected.

To avoid stalling and inadvertent abuse of this feature, I would place a hard limit of 1 second per visualization load runtime and if it exceeds that, the visualization is silently discarded so as not to impact user-experience, until the next time Projuct M is run. This ensures that artists will want to stay well-below that hard-limit and the load time of an entire visualization library wouldn't be too high over-all.

Of course, a simple user-setting could also be provided for users who want a snappier start-up time or who have weaker processing units.

If this feature is anticipated to be very popular, it could be better to play it safe and only allow a half or a tenth of a second as a load hard-limit. Otherwise we could end-up with a worst-case scenario where it takes a minute to open 100 visualizations before the first starts rendering.

Still, I would suggest a 1-second limit initially with visualizations being load in parallel (and not sequentially) to reduce overall loading time.

[kblaschke]

Since we can't really tell how long a particular preset might take loading on specific hardware, a hard limit is probably not good. If the visualizer uses Vulkan, the whole loading process can take place asynchronously in a separate thread, including loading of textures, compiling shaders and running any scripts which may generate procedural geometry and textures in advance (though for procdeural geometry, using compute and/or geometry shaders may be a better option depending on the use case).

The visualizer would then queue a preset for loading some time ahead of the actual transition point, e.g. a few seconds ahead. Then, the preset can load in the background and signal the visualizer when it's ready. The transition can then take place at the planned time, or, if the preset took longer, the visualizer could simply postpone the transition and keep playing the current preset a bit longer if required. On initial startup, the visualizer could display some kind of fast "loading" screen to not cause it to go either blank or hang.

As some background info, here's how Milkdrop mostly avoids this issue: it first loads the preset, parses/compiles the code, which is quite fast by any means. Then, Milkdrop only loads a texture or compiles a single shader per frame rendered until all preset resources are ready, then it starts the transition. This will still cause some hickups if a hugely complex shader needs to be compiled, taking more than ~20ms or so, but since older DirectX versions didn't support multi-threading just like OpenGL, this was the only way of doing it.

For shaders, there's the possibility to compile them to SPIR-V and store the binary data in addition to the shader's source code inside the preset archive to speed up loading times.

An approach similar to Valve's "Fossilize" tool might also be employed to compile the SPIR-V bytecode to actual GPU programs and cache these on disk, which would fully eliminate shader compilation at runtime once the program is cached. Note that these binaries are built specifically for a single GPU and possibly driver/firmware combination, so these files can't easily be stored in the presets. Valve actually sources those binaries from Steam users, keeping a huge database of pre-compiled shaders which Steam then downloads when needed, e.g. on the Steam Deck.