[META] Implement microphone, webcam, and other bidirectional capabilities

[ehfd]

Basically, this is the opposite direction to the audio capabilities available now.

1. Fetch microphone permissions from the web browser.
2. Receive the microphone stream and encode to opus within the browser.
3. Send the audio stream using RTP wrapped with WebRTC and the host/container Python component receives the stream.
4. The host/container Python component first decodes the opus codec, and then creates a microphone device and sends audio to the host/container with pulsesink.

From a thankful comment from @thelamer: For microphone there is really only a couple ways to skin that cat in a browser but I use signed 16 bit integers instead of float32 and block on all zeros so we don't send silence. This helps keep the mic in sync.

The gst-web components should be overhauled to accommodate new improvements such as webcams, printing, and file sharing in the future as well as to finish this job.

Other types of bidirectional media:

Webcams: The only currently viable way to expose webcams to containerized applications is to use pipewire-v4l2. Anything else needs access to kernels. This is a hard requirement unless there is an another way to not have to tinker with the kernel modules.

From a thankful comment from @thelamer: As for webcam that is just opencv and some python helpers, the thing there is having a loopback device inside the container. I suspect it is possible in userspace, but you need to port over the V4L stack to make that work or proxy the ioctls to a userspace qemu kernel or something. - Me: consider how pipewire-v4l2 works.

Printing: Implement a CUPS printing server > Convert received document to PDF/other adequate stateless format > Transport with DataChannel > Print from Web Browser
File sharing: Transport with DataChannel > Download from Web Browser

Would be convenient to investigate the following issues along the way for better usability if we are indeed overhauling gst-web for this task (Unfortunately, I don't know JavaScript too well compared to Python): #9 (AZERTY keyboard) #22 (Touch keyboard capabilities), #25 (URL parameters for authentication).

Additional issues (intertwined with gst-web and Python): #39 (Multiple users), #48 (WebTransport), #74 (WebRTC chunked DataChannel)

[ehfd]

This issue is from a request by @GeorgeDeac .

[ehfd]

Implemented support for Stereo on Opus. 1845bda

[ehfd]

Good references:

https://github.com/evshiron/parsec-web-client
https://parsec.app/blog/game-streaming-tech-in-the-browser-with-parsec-5b70d0f359bc
https://github.com/parsec-cloud/parsec-sdk

[satyaakam]

Hi , i see this issue labelled under Funding , can anyone from core developers let me know what amount of funding are you expecting in order to see this issue resolved. Need few details like amount , timelines , and process of resolving it in order to get the ball rolling .

[ehfd]

Hi! @satyaakam

Unfortunately, the core developers are currently all busy, and no cost may justify the time we are to spend on this enhancement at this time.

However, @PMohanJ, another Indian developer and junior member of the Selkies team, is hired by his corporation to develop the webcam capabilities. I suggest waiting for them to open-source this capability.

[satyaakam]

Hi @ehfd thanks for the response , I am sorry if i sounded like someone who would like to throw money or people at a problem to get it resolved :-) , I understand the way FOSS communities work , my request was more from the point of view of collaboration, thanks for pointing to @PMohanJ will be more than happy to collaborate and work towards getting it resolved .

[ehfd]

I greatly appreciate your interest in this project. @satyaakam
I would have taken the job if I had extra time.
This project competes technically with well-funded proprietary remote desktop products and conversely relies on the free time of experienced developers in very specific, but wide-spectrum technology stacks. So donations never hurt.
But currently, we really need people willing to learn the code and make their own contributions.
Bounties might work out.

[debaa98]

is there any bounty going to be paid to contributer?

[ehfd]

@debaa98 It is possible, if you are indeed interested.

[debaa98]

Yaa I am intrested

[ehfd]

But I am not the entity which issues the bounty. Check with @satyaakam.

[debaa98]

yas,i had a contact with @satyaakam