Conversion to RGB in VideoFrame.copyTo()

[Djuffin]

こんにちは TAG-さん!

I'm requesting a TAG review of Conversion to RGB in VideoFrame.copyTo().

VideoFrame.copyTo() can convert pixel data to RGB pixel format

Converting YUV video frames to RGB is often required for processing them in libraries like TensorFlow.js and OpenCV.js.
Previously the only possible way to achieve this was rendering the frame on a canvas.

Specifying VideoFrameCopyToOptions.format and VideoFrameCopyToOptions.colorSpace make it possible to asynchronously convert frames to RGB pixel format by calling VideoFrame.copyTo() without having to use an extra canvas.

• Explainer¹ (minimally containing user needs and example code): https://gist.github.com/Djuffin/9e2f98025ead49998524510cfeed8d33

• Specification URL: https://www.w3.org/TR/webcodecs/#dom-videoframecopytooptions-format

• Tests: https://github.com/web-platform-tests/wpt/blob/master/webcodecs/videoFrame-copyTo-rgb.any.js

• User research: N/A

• Security and Privacy self-review²: https://gist.github.com/Djuffin/c3742404b7c53ada227849c8b2b76b4c#file-webcodecs_contenthint_security_and_privacy-md

• Primary contacts (and their relationship to the specification):

  • Eugene Zemtsov ( @Djuffin ) Google LLC

• Organization(s)/project(s) driving the specification: Chromium

• Key pieces of existing multi-stakeholder (e.g. developers, implementers, civil society) support, review or discussion of this specification:

  • Mozilla comments: Allow converting VideoFrame to RGB pixel formats by calling VideoFrame.copyTo() mozilla/standards-positions#1017
  • WebKit comments: Allow converting VideoFrame to RGB pixel formats by calling VideoFrame.copyTo() WebKit/standards-positions#341

• External status/issue trackers for this specification (publicly visible, e.g. Chrome Status):

  • https://chromestatus.com/feature/4668827056209920

Further details:

• I have reviewed the TAG's Web Platform Design Principles
• Relevant time constraints or deadlines: couple of weeks
• The group where the work on this specification is currently being done: Media Working Group
• This work is being funded by: Google LLC

[martinthomson]

Thanks for giving this opportunity to review this work. We see this as a useful addition to the platform.

We do observe that there is a lot of uncoordinated work on color across the platform. Our understanding of the situation is that CSS is quite advanced with its work on color spaces and the media pipeline (video and so forth) is more advanced with high dynamic range content. Having a uniform platform-wide would be desirable and we'd encourage you to engage with CSS folks on the subject to have that happen. You might consider proposing a TPAC session on this topic?

We do think that the name is a little misleading here as the "copy" part of the operation isn't the most important aspect; it's the conversion or transform that is the important action. Is this something that warrants a new API? copyTo looks to be relatively new, is it possible to choose a better name?

Finally, from a process perspective, we'd strongly encourage you to take this work to the Media Working Group repo rather than sitting in someone's private GitHub account.

Either way, this work looks like a useful contribution, so we're closing this as satisfied with concerns.