Blurring of Video

[mainisameer]

This is not an issue but a implementation question we have. We want to blur the video and pass that video element to our Webrtc stream, This works well when the browser is not minimized we tried implementing the blurring using webworker but have challenges of not being able to access he video element using webworker. so in a nutshell here is where we are stuck.
If we don't use webworker and using the blurring , when the browser is minimized the frame rate drops down and the video becomes extremely lagging. when we use webworker we can not access the video element to pass that along to our SFU server.
Any thoughts on how we can implement this would be much appreciated.

[vladmandic]

why not something like this?

1. main thread grabs frame from video
2. main thread sends message containing imageData to worker using postMessage
3. worker receives message and performs blurring
4. worker sends results as message to main thread
5. main thread receives the message from worker and immediately goes back to step.1

if its something like that worker is the one controlling timing of issues and there is no throttling of the main thread because it's not running a loop, it's reacting to messages received from worker - and those are not throttled.

so the only question is how to pass video data to worker - by drawing it onto canvas, getting it's imageData and passing it as reference in postMessage call (so to avoid copy over message channel)

[puru-kohli]

so the only question is how to pass video data to worker - by drawing it onto canvas, getting it's imageData and passing it as reference in postMessage call (so to avoid copy over message channel)

This will only get image once from the video. To continuously pass the imageData to worker, will have to use setinterval which again will get throttled if chrome tab is in background

[vladmandic]

This will only get image once from the video. To continuously pass the imageData to worker, will have to use setinterval which again will get throttled if chrome tab is in background

you don't need setInterval - trigger new frame with setImmediate immediately when main thread receives a results message from the worker thread. that way you have a closed-feedback loop between main and worker threads, not a loop within either of the threads.

[mainisameer]

Let is give it a try and see what happens. Thank you for giving us feedback on the implementation question

…

On Wed, Sep 8, 2021 at 4:52 PM Vladimir Mandic ***@***.***> wrote: This will only get image once from the video. To continuously pass the imageData to worker, will have to use setinterval which again will get throttled if chrome tab is in background you don't need setInterval - trigger new frame with setImmediate immediately when main thread receives a results message from the worker thread. that way you have a closed-feedback loop between main and worker threads, not a loop within either of the threads. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#163 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQMDXSTWH52YRQECBAMHSFDUA5BOTANCNFSM5DTZKCMQ> .

[vladmandic]

btw, i've converted this issue into a discussion thread since it's not really a library issue.