Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WebXR Raw Camera Access (color) #5786

Merged
merged 15 commits into from
Nov 28, 2023
Merged

Conversation

Maksims
Copy link
Contributor

@Maksims Maksims commented Nov 4, 2023

This PR integrates Raw Camera Access API, that provides access to color image of a XR views. Along the way it introduces XrView concept which aligns with WebXR specs. It gives access to each view (single on mobile phones, and two on HMD devices), and will be extended with Depth Sensing API in the future. As current Depth Sensing API assumes a single view, so it will not work well with Quest 3's ability of providing depth maps for each view.

Test link: https://examples.playcanvas.meta.space/#/xr/ar-camera-color

Discussions: #5780

// constants
// it comes from XRView eye property
XREYE_NONE = 'none' // monoscopic screens, e.g. mobile phones
XREYE_LEFT = 'left'
XREYE_RIGHT = 'right'

// pc.XrManager
xr.views // XrViews interface that provides an access to views and their capabilities
xr.start(camera, pc.XRTYPE_AR, pc.XRSPACE_LOCALFLOOR, {
    cameraColor: true // new option to request camera color information
});

// pc.XrViews
// this class provides an access to views, and their capabilities
views.supportedColor // true if color information is supported
views.availableColor // true if color information is available.
// This information is only available after XR session has started.
// It can be supported but not available due to lack of session features request, user permissions
// or underlying system capabilities
// Quest 3 - implements necessary features, but does not expose `camera-access` to users.
views.get(eye) // returns XrView or null based on XREYE_*
views.list // an array of XrView instances
views.on('add', (view) => { }); // event when new XrView is added
views.on('remove', (view) => { }); // event when XrView is removed

// pc.XrView
// this class represents a specific view with its data
view.eye // returns a string (constants XREYE_*). 'none' - means it is monoscopic display such as mobile phone screen
view.textureColor // returns a texture with color information. If color information is not available it will be null
view.viewport // Vec4 that represents a viewport are that this view is occupying of the screen

I confirm I have read the contributing guidelines and signed the Contributor License Agreement.

@Maksims
Copy link
Contributor Author

Maksims commented Nov 14, 2023

There was a comment from @mvaligursky:

To be honest, I'd strongly prefer we use some a multi-platform solution here instead of accessing WebGL directly. That means render target and drawQuadWithShader functionality. Platform stuff is inside /platform folder only as much as possible. In the future WebGPU will add support for XR and we'd want it to just work.

I've tried multiple ways:

  1. Use RenderTargets - but it does not provide a way to create one with raw WebGLTexture, and copy method does pretty much what we do in the current implementation.
  2. Direct WebGLTexture use - this works, and is most efficient way, but it does not provide flipY, and has a rule for "opaque texture" that WebGLTexture object should not be used outside of its RAF.
  3. (Implementation in this PR) Direct WebGL functions use with blit function, that provides a way to flipY the buffer, and avoids quad a render.

Currently WebXR integration is only WebGL compatible, there are WebXR-WebGPU bindings Specs coming along, but we do not have devices that do support both APIs yet, so this is for the future PRs.

@Maksims Maksims marked this pull request as ready for review November 23, 2023 10:11
@mvaligursky
Copy link
Contributor

Fantastic PR @Maksims , thanks for this! I added few small comments, but overall this is great.

@Maksims Maksims mentioned this pull request Nov 25, 2023
3 tasks
Copy link
Contributor

@mvaligursky mvaligursky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work, thanks so much.

@mvaligursky mvaligursky merged commit 366d972 into playcanvas:main Nov 28, 2023
7 checks passed
@Maksims Maksims deleted the webxr-raw-camera branch November 29, 2023 10:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants