Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WebXR depth sensing API #2561

Merged
merged 14 commits into from
Dec 31, 2020
Merged

Conversation

Maksims
Copy link
Contributor

@Maksims Maksims commented Nov 16, 2020

Implements WebXR Depth Sensing API: https://github.com/immersive-web/depth-sensing/blob/main/explainer.md
Which is available in Chrome 87 (Beta Release) behind chrome://flags/#webxr-incubations flag.

It provides two paths (CPU and GPU) to access depth information which is provided by underlying AR system.

New APIs:

// pc.XrManager
app.xr.depthSensing // interface to access depth sensing information

// pc.XrDepthSensing
depthSensing.supported // true if depth sensing is supported
depthSensing.available // true if depth information is currently available
depthSensing.width // width of depth texture or 0 if not available
depthSensing.height // height of depth texture or 0 if not available
depthSensing.texture // texture with packed depth information
depthSensing.uvMatrix // matrix to transform depth texture to normalized space

depthSensing.getDepth(x, y); // (CPU path): get depth information using x,y in depth texture space. Returns value in meters or null if depth information is not available.

depthSensing.on('available', function() { }); // fired when depth information becomes available
depthSensing.on('unavailable', function() { }); // fired when depth information becomes unavailable
depthSensing.on('resize', function(width, height) { }); // fired when depth texture been resized, so uvMatrix needs to be updated on related materials

Test projects:

CPU Path: https://playcanvas.com/project/732030/overview/ar-depth-sensing

It implements CPU path to test for depth in middle of the screen, and allows to plant grass (like hit test example), but works way more reliably than hit test path.

GPU Path: https://playcanvas.com/project/738358/overview/ar-depth-sensing-texture

It implements GPU path by accessing depth sensing texture, and rendering it on top of the camera view, with shader that unpacks depth information and represents it as a gradient (red > yellow > green > cyan) which is distance from 0 to 8 meters.

Depth Occlusion: https://playcanvas.com/project/738703/overview/ar-depth-sensing-occlusion

Demo: https://playcanv.as/p/UN0z1XE2/ (don't forget to enable chrome://flags/#webxr-incubations)
This demo has few things: cpu path object placement, gpu path occlusion using shader and light estimation.
Link to video: https://twitter.com/mrmaxm/status/1329151944540254208

Screenshot_20201117-203525 image

I confirm I have signed the Contributor License Agreement.

@Maksims Maksims marked this pull request as ready for review November 17, 2020 19:39
@Maksims
Copy link
Contributor Author

Maksims commented Nov 18, 2020

Chrome 87 just has been released to Android. So it works behind the WebXR Incubators flag in release Chrome.

Cl seems to be failing here, with "Setup Java JDK" errors. Not sure how to fix it.

@Maksims
Copy link
Contributor Author

Maksims commented Dec 14, 2020

There have been recent changes to Depth Sensing API, based on community provided feedback.
This PR will have to be updated accordingly once changes are incorporated to the browsers.

@willeastcott willeastcott requested a review from a team December 30, 2020 14:19
@willeastcott willeastcott self-assigned this Dec 30, 2020
@willeastcott
Copy link
Contributor

Hey @Maksims, can you resolve conflicts in xr-manager.js please.

@willeastcott willeastcott merged commit e4a50dd into playcanvas:master Dec 31, 2020
@Maksims Maksims deleted the webxr-depth-sensing branch January 22, 2024 11:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: xr XR related issue
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants