Meaning and usage of viewport, size, aspect, and mouse

[mattrossman]

Any time I have to work with conversions between screen and world space, I get a little lost looking at the r3f docs. The API file states that useThree provides:

// Reactive viewport bounds, will updated on resize
const { width, height, factor, distance } = viewport

What is the intuition for what width, height, factor, and distance represent? For instance, if I have a perspective camera, the "width" that's on screen in world space will vary depending on the distance of an object. It seems like there is a fundamental relationship between these variables that's not obvious to me. It would be helpful to see some examples of common use cases for these variables.

Singularly, I see there are other variables relating to screen measurements:

size,                         // Bounds of the view (which stretches 100% and auto-adjusts)
aspect,                       // Aspect ratio (size.width / size.height)
mouse,                        // Current, centered, normalized 2D mouse coordinates

The meaning of these variables is more apparent to me, but again it would be helpful to see examples of common use cases for them which might overlap with the viewport stuff.

Edit: for reference, here's an example of how I currently do screen->world conversion. It feels like I should be doing it a different way with the variables above.

[drcmda]

viewport is screen width/height in three units. the following would fill the entire screen:

const { viewport } the  = useThree()
return (
  <mesh scale={[viewport.width, viewport.height, 1]}>
    <planeBufferGeometry />

size is the size in pixels of the dom container-

aspect is just some thing that's necessary for the internal cameras, if you had your own camera you would need to calculate it yourself. i don't know what it is for but i take it some people do - after all they did use it in all of their projects before r3f.

mouse are centered, normalized coordinates. -1 is left/top side, 0 middle, 1 is right/bottom side

distance how far the camera is away from center point.

you can edit the docs if you feel it's not entirely clear. i expect people to also add their own stuff if they need it. these were just the most relevant that i could think of.

[mattrossman]

How about the distance and factor? I get that viewport.width would cover the screen width at some distance, but how does it know what distance to compute this width at? How would I get the width at a different distance?

[drcmda]

as for your useworldcoords hook

function Ball(props) {
  const ref = useRef()
  useFrame((state) => {
    const { mouse, viewport } = state
    const { width, height } = viewport()
    ref.current.position.set((mouse.x * width) / 2, (mouse.y * height) / 2, 0)
  })
  return (
    <mesh ref={ref} {...props}>
      <sphereBufferGeometry args={[0.1, 16, 16]} />
      <meshPhysicalMaterial color="hotpink" />
    </mesh>
  )
}

it is limited though. that will only work as long as the camera is pointing from z down. and i think you know now why it works. you take the normalized mouse and just multiply it with width or half divided by two. -1 * width / 2 is left, 1 * width / 2 is right, 0 is center.

https://codesandbox.io/s/r3f-mouse-screen-to-world-forked-ybfz7?file=/src/index.js

[drcmda]

  const getCurrentViewport = React.useCallback(
    (camera: Camera = state.current.camera, target: THREE.Vector3 = new THREE.Vector3(0, 0, 0)) => {
      const { width, height } = state.current.size
      const distance = camera.position.distanceTo(target)
      if (isOrthographicCamera(camera)) {
        return { width: width / camera.zoom, height: height / camera.zoom, factor: 1, distance }
      } else {
        const fov = (camera.fov * Math.PI) / 180 // convert vertical fov to radians
        const h = 2 * Math.tan(fov / 2) * distance // visible height
        const w = h * (width / height)
        return { width: w, height: h, factor: width / w, distance }
      }
    },
    []
  )

  // Adjusts default camera
  React.useMemo(() => {
    state.current.aspect = size.width / size.height
    // Assign viewport props to the function
    Object.assign(state.current.viewport, getCurrentViewport())

    // https://github.com/drcmda/react-three-fiber/issues/92
    // Sometimes automatic default camera adjustment isn't wanted behaviour
    if (updateDefaultCamera) {
      if (isOrthographicCamera(defaultCam)) {
        defaultCam.left = size.width / -2
        defaultCam.right = size.width / 2
        defaultCam.top = size.height / 2
        defaultCam.bottom = size.height / -2
      } else {
        defaultCam.aspect = state.current.aspect
      }
      defaultCam.updateProjectionMatrix()

[mattrossman]

I was referring to z changes in the object's placement, not the camera. See how it's placed incorrectly here: https://codesandbox.io/s/r3f-mouse-screen-to-world-forked-jifi8

The unproject approach I was using allows you to set the z distance at which the object will be placed, while still staying in the ray of the cursor. It's also really unintuitive though, so if there is a simple way to compute this using the distance and factor, it would be preferable.

Edit: unintuitive, not intuitive :P

[drcmda]

ah, yes, ofc. could you look into the sources real quick? this would be a great addition for the events. you hover over something and you get the unproject. i have something like that already, but the math is probably off.

[mattrossman]

I finally found a solution. There's no need to unproject because the default raycaster already provides the vector I want in racaster.ray.direction. Similarly, there's a bunch of helper functions like raycaster.ray.at() for getting the point at some distance along the cursor ray (which forms a sort of semi-circle around you), or raycaster.ray.intersectPlane() for calculating points on a plane at the desired distance.

The simplest example of what I want can be condensed down to:

useFrame(({ raycaster }) => {
  raycaster.ray.at(distance, ref.current.position)
})

https://codesandbox.io/s/r3f-mouse-to-world-elh73?file=/src/index.js
(Click to change the dot distance, see how it remains in the line of sight)

This approach is robust to camera movement and rotation with OrbitControls. You can improve the result by checking for intersections along a plane that follows the camera orientation. That can lead to some really clean dragging behavior:
https://codesandbox.io/s/react-three-draggable-cxu37?file=/src/App.js

[pedroalmeida415]

It works wonders! Thanks for this @mattrossman, real nice work