Modulation Discovery with Differentiable Digital Signal Processing

Christopher Mitcheltree, Hao Hao Tan, and Joshua D. Reiss

Abstract

---

Modulations are a critical part of sound design and music production, enabling the creation of complex and evolving audio. Modern synthesizers provide envelopes, low frequency oscillators, and more parameter automation tools that allow users to modulate the output with ease. However, determining the modulation signals used to create a sound is difficult, and existing sound-matching / parameter estimation systems are often uninterpretable black boxes or predict high-dimensional framewise parameter values without considering the shape, structure, and routing of the underlying modulation curves. We propose a neural sound-matching approach that leverages modulation extraction, constrained control signal parameterizations, and differentiable digital signal processing (DDSP) to discover the modulations present in a sound. We demonstrate the effectiveness of our approach on highly modulated synthetic and real audio samples, its applicability to different DDSP synth architectures, and investigate the trade-off it incurs between interpretability and sound-matching accuracy. We make our code and audio samples available and provide the trained DDSP synths in a VST plugin.

Citation

---

Accepted to the IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), Lake Tahoe, CA, USA, 12 - 15 October 2025 (best paper candidate).

@inproceedings{mitcheltree2025modulation,
    title={Modulation Discovery with Differentiable Digital Signal Processing},
    author={Christopher Mitcheltree and Hao Hao Tan and Joshua D. Reiss},
    booktitle={IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)},
    year={2025}
}

Instructions for Reproducibility

---

1. Clone this repository and open its directory.
2. Install the requirements:
   conda env create --file=conda_env.yml
   or
   conda create -n md python=3.10
conda activate md
pip install uv
uv pip install 'torch==2.2.2' lightning numba torchaudio torchvision auraloss black librosa matplotlib numpy nvitop pandas schedulefree 'scipy<=1.11.4' tensorboard wandb pip-chill 'numpy<2.0.0' 'jsonargparse[signatures]' torchlibrosa shapely hypy_utils encodec transformers nnaudio resampy jsonschema
uv pip install msclap --no-deps
uv pip install 'git+https://github.com/Neutone/neutone_sdk.git' --no-deps
   For posterity, requirements.txt and requirements_all.txt are also provided.
3. The source code can be explored in the mod_discovery/ directory.
4. All experiment config files can be found in the configs/ directory.
5. Create an out directory (mkdir out).
6. All experiments can be run by modifying scripts/train.py and the corresponding configs/.../train_ ... .yml config file and then running python scripts/train.py.
   Make sure your PYTHONPATH has been set correctly by running commands like:
   export PYTHONPATH=$PYTHONPATH:BASE_DIR/mod_discovery/,
   export PYTHONPATH=$PYTHONPATH:BASE_DIR/torchlpc/,
   and export PYTHONPATH=$PYTHONPATH:BASE_DIR/fadtk/.
7. Exported Neutone models of the learned synths can be found in the docs/neutone/ directory. They can be loaded and run in the user's preferred DAW through the Neutone FX host plugin VST .
8. The source code is currently not documented, but don't hesitate to open an issue if you have any questions or comments.