fluidsynth and pulseaudio with xrdp sink

[rayrapetyan]

I have a VM, it has pulseaudio with xrdp and xrdp sink configured. Sound works fine in this setup through RDP. Now I'm trying to play a midi file, but I'm missing something in configuration.

Here is what I've tried:
1.

fluidsynth --server --no-shell --audio-driver=pulseaudio /usr/share/sounds/sf3/default-GM.sf3
FluidSynth runtime version 2.1.7
Copyright (C) 2000-2021 Peter Hanappe and others.
Distributed under the LGPL license.
SoundFont(R) is a registered trademark of E-mu Systems, Inc.

ALSA lib seq_hw.c:466:(snd_seq_hw_open) open /dev/snd/seq failed: Permission denied
fluidsynth: error: Error opening ALSA sequencer
Failed to create the MIDI thread; no MIDI input
will be available. You can access the synthesizer 
through the console.
fluidsynth: Using PulseAudio driver
fluidsynth: warning: Failed to set thread to high priority

There is no fluid output in aconnect:

sudo aconnect -l
[sudo] password for gamer: 
client 0: 'System' [type=kernel]
    0 'Timer           '
    1 'Announce        '
client 14: 'Midi Through' [type=kernel]
    0 'Midi Through Port-0'

But it's visible to pulseaudio:

pactl list sink-inputs
Sink Input #16
	Driver: protocol-native.c
	Owner Module: 5
	Client: 46
	Sink: 1
	Sample Specification: float32le 2ch 44100Hz
	Channel Map: front-left,front-right
	Format: pcm, format.sample_format = "\"float32le\""  format.rate = "44100"  format.channels = "2"  format.channel_map = "\"front-left,front-right\""
	Corked: no
	Mute: no
	Volume: front-left: 65536 / 100% / 0.00 dB,   front-right: 65536 / 100% / 0.00 dB
	        balance 0.00
	Buffer Latency: 1337 usec
	Sink Latency: 201 usec
	Resample method: copy
	Properties:
		media.name = "FluidSynth output"
		application.name = "FluidSynth"
		native-protocol.peer = "UNIX socket client"
		native-protocol.version = "34"
		media.role = "music"
		application.process.id = "3585"
		application.process.user = "user"
		application.process.host = "myservice"
		application.process.binary = "fluidsynth"
		application.language = "C"
		window.x11.display = ":10.0"
		application.process.machine_id = "e8a4b59ac0c40c0bd502574421e3d4ad"

How can I feed a midi file to fluidsynth here? aplaymidi requires a port from aconnect -l output, but as we can see it's not there. Thanks.

[mawe42]

Fluidsynth actually tells you why there is no ALSA MIDI port:

ALSA lib seq_hw.c:466:(snd_seq_hw_open) open /dev/snd/seq failed: Permission denied
fluidsynth: error: Error opening ALSA sequencer
Failed to create the MIDI thread; no MIDI input
will be available. You can access the synthesizer 
through the console.

It seems your user doesn't have the required permissions to open /dev/snd/seq. On some distros you need to be in the audio group to be able to do that.

[rayrapetyan]

But /dev/snd/seq is a real device, let's assume there is no sound card yet on the host machine. Is there a way to make fluidsynth to work with pulseaudio, which redirects all sounds into sinks and works perfectly well without a sound card?

[mawe42]

/dev/snd/seq belongs to the ALSA sequencer, it's not a soundcard. Above you used sudo in front of your aconnect call (which also uses /dev/snd/seq). So either start fluidsynth with sudo as well or simply check the file permissions on /dev/snd/seq and add your user to the relevant group (probably "audio").

[mawe42]

And if /dev/snd/seq is passed into the VM from your host and you don't want to rely on the host device, then change the VM config so that the device isn't passed to the VM and do a modprobe snd-seq in your VM to get a "local" sequencer.

[rayrapetyan]

Oh you're right, it's not a physical device, adding user into audio group resolved the issue, thanks!
The only weird thing I can observe is the CPU usage in "idle" mode by pulseaudio (~20%) and fluidsynth (~12%) similar to the issue here: #338.

pulseaudio constantly performs:

ppoll([{fd=3, events=POLLIN}, {fd=10, events=POLLIN}, {fd=24, events=POLLIN}, {fd=4, events=POLLIN}, {fd=11, events=POLLIN}, {fd=22, events=POLLIN}, {fd=18, events=POLLIN}, {fd=21, events=POLLIN}, {fd=14, events=POLLIN}, {fd=17, events=POLLIN}, {fd=7, events=POLLIN}], 11, NULL, NULL, 8) = 1 ([{fd=17, revents=POLLIN}])
read(17, "\1\0\0\0\0\0\0\0", 8)         = 8
write(5, "W", 1)                        = 1
read(3, "W", 10)                        = 1
write(25, "\1\0\0\0\0\0\0\0", 8)        = 8
read(3, 0x7fff475cfede, 10)             = -1 EAGAIN (Resource temporarily unavailable)

Meanwhile fluidsynth is locked in:

futex(0x7ff4737fe9d0, FUTEX_WAIT, 1056, NULL

Somehow fluidsynth is causing that, because when fluidsynth process is not running, pulseaudio consumes nothing.

[mawe42]

FluidSynth has no "idle" state and is multithreaded. The audio driver runs in it's own thread and continuously renders audio. If there is no MIDI input it simply renders silence. I don't know anything about your device and VM specs, so it's difficult to judge if 20% PA and 12% FluidSynth is higher than expected. But as you are using PulseAudio to send the audio via RDP to another "machine", I would definitely expect some noticeable load.

[rayrapetyan]

It's really much higher than expected. E.g. when fluidsynth is not running and there is a non-midi source producing sounds pulseaudio uses about 2% of CPU (with spikes to 3%) on the same xrdp-sink and networking stack. I'll try to debug more.

[rayrapetyan]

Here are some collected metrics (timeout 10s strace -c -p PID_OF_PULSEAUDIO)

1. fluidsynth is down, no audio sources

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 86.90    0.003603         514         7           ppoll
  7.12    0.000295          14        21           write
  5.84    0.000242          11        21         7 read
  0.14    0.000006           6         1         1 futex
------ ----------- ----------- --------- --------- ----------------
100.00    0.004146          82        50         8 total

2. fluidsynth is down, one active sound producer (browser)

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 44.76    0.011721          24       478           ppoll
 28.77    0.007533           6      1250       478 read
 26.00    0.006808           8       759           write
  0.47    0.000123          17         7         4 futex
------ ----------- ----------- --------- --------- ----------------
100.00    0.026185          10      2494       482 total

3. fluidsynth is up, no sound producers

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 39.82    0.696119          10     69456           write
 36.51    0.638188           9     65966     25282 read
 21.35    0.373281          14     25284           ppoll
  2.32    0.040605          57       709       320 futex
------ ----------- ----------- --------- --------- ----------------
100.00    1.748193          10    161415     25602 total

[rayrapetyan]

Looks like --sample-rate affects the CPU usage significantly, the above was for 48000, for 8000 CPU usage is dropping to 4% and 8% respectively.

[mawe42]

Yes, sample rate heavily affects FluidSynths CPU usage, as do the reverb and chorus effects (disable with -R0 and -C0) and also the period size (set to a higher value if you are only playing MIDI files and don't care about real-time, e.g. -z4096).

[mawe42]

You might get away with just setting the period size to the maximum: -z8192, that should reduce your CPU load significantly. FluidSynth uses a period size of 64 by default, which is great for real-time playback but quite hard on the CPU.

[rayrapetyan]

Thanks, -z8192 alone reduced CPU usage by fluidsynth down to 1.3%... Now everything looks awesome!