Sharing my audio visualizer plugin #984
D0rk80
started this conversation in
Show and tell
Replies: 1 comment
-
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment




Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Sharing my audio visualizer plugin — would love your feedback.
I built a custom WebGL2 audio visualizer engine inside a WordPress plugin and wanted to share it with someone who would appreciate the technical side. It has grown considerably since I started — 133 templates now across 2D Canvas, Three.js, and raw WebGL2 — so I will try to cover the parts that are actually interesting rather than just listing features.
The core engine
It is a custom WebGL2 renderer called MV — not Three.js with a shader layer on top, but raw WebGL2 with its own shader compiler, GLSL chunk dependency resolver, geometry factory, material system, FBO manager, camera system, and audio analysis pipeline. About 3,500 lines in a single shared file. 133 templates run on top of it.
We got 129 of those templates fully off Three.js. The remaining 4 are a boid flocking system, an L-system tree, an orbital mechanics simulation, and a Canvas2D ripple — things that genuinely need a rewrite rather than a wrapper swap. Everything else runs raw WebGL2.
Two sub-systems worth calling out
MV.ortho is a lightweight fullscreen shader runner we built to replace Three.js for 2D templates. It is about 200 lines — creates a WebGL2 context, compiles your fragment shader, uploads the full v5 audio uniform set automatically each frame (uBass, uMid, uTreble, uGroove, uBeatPhase, uBarPhase, uPitchHue, uStereoW, uTrans, and a 256-bin uFreqTex texture), handles ping-pong FBOs for feedback effects, and calls your onFrame callback for anything extra. Migrating a Three.js fullscreen shader template to it is about 10 lines of change.
MV.milk is a .milk preset file loader we built after reading through your source. It parses the key=value format, assembles the multiline code blocks, runs a JavaScript HLSL to GLSL transpiler (float2/3/4, tex2D, lerp, saturate, frac, atan2, mul, semantic stripping, shader_body entry point expansion), evaluates per-frame equations in a sandboxed new Function() context (handling if(), above(), below(), sqr(), sigmoid(), band(), etc.), and maintains q1-q32 state that flows into the shaders via packed _qa-_qh uniforms matching your PresetShaderHeaderGlsl330.inc exactly. Two-pass render: warp to ping-pong FBO, composite to screen. The UI is a drag-and-drop zone — drop any .milk file and it runs.
The transpiler handles 90% of presets. What it does not handle: the full HLSL AST cases your hlslparser deals with, sampler_state{} blocks properly, and a few edge cases around swizzle assignment. But for the standard shader_body + tex2D + float3 ret pattern it produces valid GLSL300 on the first try.
We also fixed a systemic issue in MV.milk since the first version: the wave inject ring was drawing after the warp pass, meaning it bypassed the FBO clamp and accumulated unbounded on high-decay presets. Reordered to inject, warp (clamp), composite. Also switched feedback FBOs from LINEAR to NEAREST filtering — LINEAR compounds a sub-pixel blur 60 times per second through zoom/rotation transforms, which degraded sharpness noticeably over the first few seconds of playback. NEAREST keeps the crisp initial look indefinitely.
Templates worth looking at technically
The reaction-diffusion (Gray-Scott) runs 4-8 simulation steps per frame in 512x512 float ping-pong FBOs (2048x2048 on Ultra tier), feed/kill rates driven by bass and mid, morphing between coral, spot, and worm morphologies. The Navier-Stokes fluid does velocity diffusion, divergence-free pressure solve, and dye advection — same architecture as the fluid sim in MilkDrop but without the approximations our milk ports use. The cymatics template puts actual Chladni math in the vertex shader — cos(mpix)cos(npiy) - cos(npix)cos(mpiy) — five modes superimposed, each driven by a frequency band, sand particle points accumulated at nodal lines. It got a significant pass since first version: 8k particles now with a mass attribute for size variety, a second fullscreen pass rendering the Chladni node lines themselves as glowing geometry, more responsive physics with adaptive friction and radial kick scatter.
The reflections template does real two-pass planar reflection: scene renders from a Y-flipped camera into an FBO, the floor samples that FBO with Fresnel blend, kick triggers a shockwave ripple on the UV offset. Getting this working correctly took longer than I would like to admit. The bug that was invisible for weeks: the floor fragment shader was sampling the reflection FBO using clip-space coordinates from the main camera, but the FBO was rendered from the reflected camera. Two different projection matrices, completely mismatched UV coordinates, zero reflection visible. The fix is screen-space — project the floor pixel through the main camera to get its screen position, flip Y, sample there. The reflection of screen position (x, y) lives at (x, 1-y) in the reflected camera's render. Once that clicked everything else worked immediately.
Five kaleidoscope-family templates use SDF raymarching with folding-space symmetry. The most technically interesting is Geometric Dream — an Apollonian gasket fractal via sphere inversion IFS. The scene is computed by repeatedly folding and scaling space, which produces infinite nested sphere geometry. Per-iteration hue cycling gives the characteristic rainbow layering, and bass modulates the fold scale to morph the fractal structure in real time. It produces results that look computationally impossible at 60fps.
Other recent additions: Aurora Tunnel (volumetric raymarched aurora curtains in a hexagonal corridor), Magnetic Storm (50k particles tracing 2D dipole field lines from orbiting poles, polarity flips on kick), Holographic Grid (sci-fi HUD — perspective grid floor, floating frequency rings, CRT scan lines, glitch artifacts), Smoke Tendrils (curl-noise smoke simulation using divergence-free velocity from FBM curl), Clifford Attractor (60k particles iterating through the strange attractor, A/B/C/D parameters morphed by audio), Neon Grid (raymarched infinite city with SDF box buildings, frequency-reactive neon edge glow, Fresnel reflective ground), Liquid Metal (raymarched mercury metaballs via smooth-min SDF, iridescent PBR with Schlick Fresnel), and Bar Reflect 3D (64 rainbow capsule bars with hemisphere dome caps over a glossy black piano-lacquer floor with working planar reflections and per-bar coloured caustic pools).
GPU quality tier system
We added automatic GPU detection via WEBGL_debug_renderer_info. It pattern-matches the renderer string and assigns one of four tiers (low/medium/high/ultra). An RTX 4070 hits ultra: 256 raymarch steps at full native resolution, 2048px fluid FBOs, 50,000 instanced particles, 2048x2048 milk FBOs. Lower tiers scale everything down proportionally. There is also a manual quality selector in the UI that persists to localStorage.
Audio analysis
The pipeline produces 46+ named fields per frame: 256-bin Bark-scale A-weighted perceptual buffer, kick/snare envelopes, beat phase, bar phase, BPM with confidence, groove regularity score, per-band transients (8 bands), spectral centroid, harmonic ratio, stereo width and pan, and HPS pitch detection with MIDI note and octave. MV.freq.sample(mv, 0.5) hits around 1kHz — the peak of the A-weighting curve — not the midpoint of a linear bin array.
Material system
The material system was completely rewritten. All 21 original surface shaders were rebuilt with proper physically-based lighting — GGX specular, Schlick Fresnel, roughness parameters — replacing what were mostly 2-3 line HSL tints. Six new materials were added: Carbon Fibre (woven anisotropic pattern with two perpendicular fibre directions and correct conductor reflectance), Ceramic (rough glossy coat over SSS-dominant diffuse for thin-wall translucency), Mercury (near-perfect mirror with noise-driven liquid surface perturbation), Holo Foil (three stacked iridescent layers at different diffraction frequencies), Bioluminescent (pulsing FBM cell noise driving SSS and emissive glow simultaneously), and Wax (SSS-dominant — light sinks deep before scattering, waxy Fresnel rim coat). Total now 27 surface styles, 28 material presets. Swapping materials on the same template gives genuinely different lighting — not just colour changes.
How it is packaged
Gutenberg block. Drop it in a page, upload audio or enter a stream URL, pick a template from the searchable dropdown, pick a material. Auto-cycle mode, per-template settings, admin playlist manager. The whole plugin including 133 templates, the engine, audio analysis, and the milk loader is 830KB zipped.
Demo at https://mspservices.us/services/resources/music-visualizer/ — happy to answer questions on any of the technical side, or go deeper on anything above.
Quick update since the last message:
The engine just got a significant internal upgrade. The shader chunk library grew from 56 utility chunks to 79 — additions include proper multi-layer SSS (epidermis/dermis separation with forward scatter), reusable anisotropy, clearcoat, transmission/refraction, procedural IBL, volumetric fog and god rays, SSAO, cel/toon shading, pen-ink hatching, full atmosphere scattering, animated water normals, volumetric flame, expanded SDF primitives with smooth boolean ops, area/spot/hemisphere lights, curl noise particle dynamics, terrain generation with erosion, and lens flare with barrel distortion.
Two new engine systems were added. MV.tex is a texture management layer with image loading, CPU-side procedural generation (Perlin, FBM, marble, rust, wood, normal maps, checker), GPU-side normal map baking from height textures, and a bind/cache API. MV.post is a unified post-processing pass system covering two-pass Kawase bloom, ACES/Reinhard/Filmic tonemapping, chromatic aberration, film grain, vignette, a full CRT effect with barrel distortion and phosphor mask, and lift/gamma/gain colour grading — all callable as single-line functions from any template.
MV.geo gained heightmap mesh generation, surface of revolution (lathe), and proper capsule geometry with hemisphere caps. Total engine size is now 5,300 lines. Everything is still one file, plugin still under 850KB.
Quick update — engine now at v8.18.36 Template count is up to 180+ since the last post. Added about 50 new ones across a few categories: Geometric wireframes — Octahedron, Icosahedron, Dodecahedron, Merkaba (star tetrahedron), Star Cage, Hyper Prism, Nested Spheres, Torus Knot — all using the same billboard-edge + vertex-orb system as the Tesseract. Instanced glowing edges that react to frequency bands, ghost trails, orbit camera. Ortho fragment shaders — Tunnel Rush, Worm Hole, Plasma Storm, Crystal Fractal, Hypno Spiral, Lava Flow, Galaxy Arms, Neon Rain, Retro Grid (synthwave perspective grid + sun + stars), Audio Eye (anatomical iris that dilates with bass), Pulse Web, Storm Eye, Mandala Pulse, and about 15 more. Redesigns — Light Spires 3D got a full rebuild: 48 hexagonal crystal prisms in 3 concentric rings, custom tapered geometry with pyramid tips, GGX lighting with inner glow traveling up the crystal, caustic floor pools per spire, bloom FBO. Cinematic Bars replaced the star background with 300 3D ember particles rising from the floor. Galaxy Rings fixed — rings were LINE_LOOP which WebGL2 caps at 1px wide, switched to point sprites so they're actually visible. Performance — Quantum Wave was doing ~720,000 trig calls per frame in a JS pixel loop (140ms/frame). Rewrote it as a WebGL2 fragment shader, same visual, now sub-1ms. Template selector — replaced the single-column 220px dropdown with a 580px panel: category sidebar on the left (12 categories, counts, active highlight), 3-column tile grid on the right, live search. Categories: Bars & Spectrum, Waveform, 3D Shapes, 3D Scenes, Aurora & Sky, Particles & Fluid, Organic & Nature, Abstract Shader, Geometric 2D, Classic & Retro, Milkdrop, Experimental. Plugin is still under 1MB zipped.
Quick update — engine now at v8.18.59. Template count is up to 227.
Audio pipeline overhaul — the root cause of weak reactivity
The previous version had a systemic problem with uBass, uMid, uTreble, and uEnergy. These were sourced from fastBands, which normalizes against longAvg3 — a long-running average initialized to 1.0 with a decay rate of 0.992 per frame. At 60fps that takes over a minute to normalize. For the first minute of playback, uBass was effectively 0.05–0.15 regardless of what was actually playing. Templates appeared unresponsive. After normalization settled, values sat near 0.5 at typical volume with almost no dynamic range between quiet and loud passages.
The fix: uBass/uMid/uTreble/uEnergy now derive directly from freqP bin averages. Bass = sum of bins 0–20 (20–300Hz on the Bark scale) divided by 20, scaled by 2.0. Mid = bins 20–70, same treatment. Treble = peak of bins 70–160 rather than mean (treble bins are sparse — mean underestimates badly). Energy = mean of all 180 active bins. These are immediate, require no warm-up, and give a genuine 0 → 0.4 → 1.0 range from silence to loud peak. The secondary problem was that freqP bins at steady state normalize to ~0.4 each, so a multiplier of 3.5 (previous calibration) was pegging typical music at 1.0 with no headroom. Reduced to 2.0 for bass/mid so typical level maps to ~0.40 and a loud bass hit reaches ~0.85.
Three new audio uniforms added to the standard ortho set: uSnare, uSnareEnv (smoothed snare envelope, same architecture as kick), and uHiHat (derived from the high-band transient). uStereoL and uStereoR are now computed from stereoPan + stereoWidth and uploaded each frame — templates can visually split the left and right channels independently.
Bug sweep across 90+ templates
Running a diagnostic script across all 227 templates exposed several categories of silent bugs:
texture(uFreqTex, ...).r / 255.0 in six templates including Piano Keys. GLSL texture() already returns 0..1. Dividing by 255 made every key brightness effectively zero — that was the entire reason Piano Keys showed no movement at all.
uBeatPhase used as a flash multiplier in 42 templates. uBeatPhase is a smooth 0→1 ramp across the beat duration — using it as col += glow * uBeatPhase * 4.0 produces a slow fade, not a snap. Replaced with uBeatEnv in all 42, which edge-detects the beat transition and decays exponentially in 180ms.
High base values drowning out audio in 11 templates. Pattern: col += glow * (0.6 + uEnergy * 0.4). With base 0.6, the audio contribution is only 40% of the total — at silence the template is already at 60% brightness, and loud peaks only reach 100%. Rebalanced to (0.1 + uEnergy * 1.2) or similar throughout, so silence is actually dark.
Linear frequency sampling in 8 templates. Bark-scale correction (pow(freq, 0.6)) added to uFreqTex lookups so bass frequencies map to bass bins rather than the bottom 5% of the texture.
uBeatEnv injected into non-ortho shader GLSL without being declared as a uniform — caught from browser console errors on circular3d, orbitalRings, plasma, and about 15 others. The beatPhase sweep was too broad. Fixed: non-ortho templates reverted to their existing uBeatPhase or uKick uniforms which they already upload.
MV.gl.clear() not setting the viewport. 69 templates using MV.gl.init had no gl.viewport() call per frame — correct on load but broken after any window resize. Fixed in the engine: MV.gl.clear() now always syncs the viewport to canvas.width/height before clearing, covering all 69 in one line.
New GLSL chunks — library now at 115
Upgraded four existing chunks substantially: fbm gained mvFbmN (configurable octave/lacunarity/gain), mvRidgedFbm (sharp ridge output instead of smooth bumps), and mvWarpedFbm (Inigo Quilez double domain warp — the one that produces genuinely organic shapes). volumetric rebuilt with Henyey-Greenstein phase scattering, proper layered height fog, and mvMarchClouds (full volumetric cloud raymarcher with per-step sun shadowing). ssao upgraded to 16-sample hemisphere with golden angle spiral distribution and per-pixel random rotation. atmosphere now does full Rayleigh + Mie with wavelength-dependent sky colour, proper horizon haze, mvTwilight for day/night blending.
New chunks: ggx (proper GGX D, Smith G, full mvGGXSpec lobe, mvPBRFull all-in-one — pbr2 now depends on it), sdfOps (full SDF primitive library plus smooth unions/subtractions, domain repetition, polar repeat, twist, bend), tonemapping (ACES, Reinhard, Filmic, PQ), colorGrading (lift/gamma/gain, saturation, contrast, temperature, mvColorGrade all-in-one), shadowRay (soft shadow marching + 5-tap SDF AO), godRays2 (Crytek screen-space shafts + 3D volumetric shaft marcher), scanline2, vignette2, bloom2. All nine surface style chunks now depend on tonemapping and colorGrading.
New and rebuilt templates — 47 added since last post
Audio-reactive stereo/transient split: Stereo Field (L/R frequency bars that physically diverge as stereo width increases, pan indicator ball), Stereo Mirror (FBM noise field that diverges left/right as stereo width increases — mono content is symmetric, wide stereo pulls the two halves into different patterns), Transient Storm (three completely separate visual layers: kick fires ground shockwave rings, snare fires radial 12-point burst, hi-hat drops sparkle rain from top — all independent and simultaneous), Piano Keys (88-key piano A0–C8, each key brightness from its Bark-mapped FFT bin, HPS pitch detection glows the detected note and harmonics, snare = mid-register white flash, hi-hat = high-key shimmer).
Spectral analysis: Spectral Waterfall (proper ping-pong FBO implementation — each frame writes a new FFT row at the top, shifts the entire texture down one row, Bark-scale frequency mapping, pitch detection marker, beat row flash).
Volumetric: Cloud Scape (volumetric cloud flythrough using mvMarchClouds — HG scattering, per-step sun shadowing, Rayleigh atmosphere, ACES tonemap), God Ray Scene (multiple light shafts with HG phase scattering, dust particles, stereo-panned sources).
New ortho shaders: Frequency Terrain (FFT data rendered as a perspective wireframe landscape with proper ray-to-plane projection in pure GLSL), Plasma Ball (electric sphere with bass-driven animated tendrils via FBM, HG-like forward scattering), Chromatic Wave (RGB-split waveform with CRT scanlines — each colour channel offset by a different frequency band), Neon Grid V2 (city flythrough with GGX wet ground reflections and ACES tonemap).
Upgraded from Canvas2D to WebGL: Oscilloscope, Neon Spiral, Star Burst — all three rebuilt as ortho fragment shaders with Bark-scale frequency sampling, palette support, and beat envelope. The Oscilloscope now supports a Lissajous mode (toggled by groove) where X and Y axes are driven by different frequency ranges.
Minimal/elegant counterpoint: Minimal Bars (64 clean bars, Bark scale, thin accent line at bar top, near-black bg — designed for streaming page embeds), Glow Line (single luminous waveform with soft bloom, no effects), Breathe (7 slow-expanding rings, meditative, designed for ambient/classical), Ink Stroke (brushstroke waveform on paper texture with ink fiber noise and kick splatter), Constellation (40 stars with frequency-reactive connecting lines and twinkle).
Rewritten for reactivity: Atomic Orbit completely rebuilt — proper Gaussian radial distributions centered at each orbital radius, s/p/d angular lobe shapes, three orbiting electron dots with band-driven speed, nucleus that blazes white on kick.
Milk loader improvements
Transpiler now handles sampler_state{} block stripping, SamplerState declarations, swizzle l-value assignments (ret.rgb = x rewritten to component assigns since GLSL 300es doesn't allow swizzle l-values), sincos(), asfloat/asint/asuint, storage qualifiers (nointerpolation, row_major), C-style casts ((float)x → float(x)). Coverage now approximately 97% of common presets.
UI additions
BPM display in the now-playing bar — appears when BPM confidence exceeds 35% and detected tempo is above 40, fades in smoothly. Custom palette color picker — three-dot button opens a floating panel with three color stops, 2-stop/3-stop mode toggle, live gradient preview, Apply sets window.MV_PALETTE immediately. Screenshot button (P key) — toDataURL('image/png'), filename from current track name. Five-style template transition rotation: fade, zoom, scatter (blur+brighten), ripple (hue-rotate blur), slice (skew slide) — cycles on each template switch. Mobile touch gestures: swipe left/right = prev/next track, swipe up = fullscreen, long-press 600ms = screenshot.
Plugin is still under 1.1MB zipped.
v8.18.59 → v8.19.1
~236 new templates added across all categories (Aurora & Sky, Particles & Fluid, Organic & Nature, Abstract Shader, Geometric 2D, Classic & Retro, Milkdrop, Experimental, 3D Shapes, 3D Scenes) bringing the total to 465 templates.
crystalCave3d fully rewritten — proper hexagonal prism crystals in concentric rings, IOR-style facet shading, bioluminescent tip glow, reflective liquid floor with beat-driven ripple rings, additive glow sprites.
Engine fixes — drawInstanced overflow guard, !_ok program link check on draw calls, bloom=0 black fix (mat.bright*(0.5+mat.bloom*0.5)).
CSS — viewport max-height fix for ultrawide displays, palette dropdown sizing.
PHP transient cache now auto-flushes on version change so new templates appear immediately after a zip upload without needing to deactivate/reactivate the plugin.
Bug fixes — unquoted PHP array constants crashing WordPress on load, missing comma parse error, 155 templates registered in JS files but absent from the PHP flat array (causing "Template not found").
Beta Was this translation helpful? Give feedback.
All reactions