Samsung Android - WebXR AR capture without camera-access: FOV mismatch when compositing getUserMedia

🌵 Needle Engine
1

Hi,

I’m trying to implement AR photo capture on Samsung Galaxy Android
(Chrome) without using camera-access feature, because it causes
GPU crash on Samsung devices.

My approach:

  1. getUserMedia + ImageCapture.takePhoto() for camera background
  2. Offscreen WebGLRenderer to render 3D scene separately
  3. Composite camera photo + 3D scene on canvas
  4. Save as JPEG

The problem:
The captured 3D scene appears zoomed in compared to
what the user sees in AR. The camera background looks correct,
but the 3D objects are much larger/closer than in the actual AR view.

My current setup:

  • renderer.domElement: 1080 x 2340 (devicePixelRatio: 2.625)
  • xrCam cameras[0] projectionMatrix fovY: 73.9, aspect: 0.462
  • mainCamera.fov: 50
  • takePhoto() resolution: 3060 x 4080

I’ve tried:

  • Copying xrCam.cameras[0].projectionMatrix directly → wrong
  • Using fovY from projectionMatrix (73.9) → zoomed in
  • Using mainCamera.fov (50) → still zoomed in
  • Various FOV multipliers → either zoomed or distorted

Questions:

  1. What FOV should I use for the offscreen renderer to match
    the actual AR view?
  2. Is there a way to get the exact FOV that Needle Engine uses
    internally for WebXR rendering?
  3. Is there a better approach for AR capture on Samsung Android
    without camera-access?

Thank you

2

The FOV mismatch is because you’re extracting fovY from the projection matrix but not accounting for the full matrix transformation. The XR camera’s projection matrix already defines the correct frustum — just copy it directly to your offscreen camera instead of computing separate FOV.

const xrCam = renderer.xr.getCamera();
const offscreenCam = /* your offscreen perspective camera */;
offscreenCam.projectionMatrix.copy(xrCam.cameras[0].projectionMatrix);

Also ensure your offscreen renderer’s size matches the display aspect ratio (not the takePhoto resolution). Your takePhoto() returns 3060x4080 (portrait), but the AR view uses 1080x2340 (also portrait, but different ratio). Scale the 3D render to the same aspect as the captured photo by setting the offscreen canvas size to match the photo dimensions, then use the projection matrix from the XR camera.

On Samsung, the GPU crash is a known issue when enabling camera access in WebXR. Your workaround is valid — but if the matrix copy still gives zoom, check that you’re using xrCam.cameras[0] (the eye used for rendering). Share a URL if you want me to dig deeper — otherwise this should fix the zoom.

Sources: Needle Engine Docs (66%), Needle Engine Docs (62%)

1 replies left - Become PRO for more replies.
Reply to this post directly if you need further help (bottom right button)

3

13
13353×283 47.1 KB

The first image is the actual screen
The second image was captured and downloaded.

It’s zooming in like this and getting distorted, how do I solve it?

4

The provided context doesn’t contain a direct solution for your AR zoom issue. However, the problem is generic: your offscreen camera’s projection doesn’t match the XR camera’s. Here’s the fix:

  1. Copy the XR camera’s projection matrix directly into your offscreen camera (don’t extract FOV manually).
    offscreenCamera.projectionMatrix.copy(xrCam.cameras[0].projectionMatrix);
  2. Set your offscreen renderer size to match the photo resolution (3060×4080) but preserve the aspect ratio of the XR view:
    offscreenRenderer.setSize(3060, 4080);
offscreenCamera.aspect = 3060 / 4080;  // matches photo aspect
offscreenCamera.updateProjectionMatrix();
  3. Ensure you’re using the correct XR camera for the current view (left vs right eye) – xrCam.cameras[0] is correct for monoscopic AR.

If this still doesn’t work, share a minimal reproduction URL — that’s the fastest way to get a precise answer.

Sources: Needle Engine Docs (65%), Needle Engine Docs (64%)

0 replies left - Become PRO for more replies.
Reply to this post directly if you need further help (bottom right button)

5

Hi,

Following up on my previous question about AR capture on Samsung Android.

I’m compositing getUserMedia photo + offscreen 3D render.
Projection matrix copy works (objects appear), but 3D is still
zoomed in compared to actual AR view.

Current setup:

  • renderer.domElement: 1080 x 2340 (devicePixelRatio: 2.625)
  • xrCam cameras[0] fovY: 73.9, aspect: 0.462
  • mainCamera.fov: 50, aspect: 0.511
  • takePhoto() resolution: 3060 x 4080
  • WebARSessionRoot scale: [1,1,1], position: [0,0,0]
  • userScale: undefined

What I’m doing:

  1. onAfterRender() - save xrCam pose every frame:
    _lastXRCamPos, _lastXRCamQuat from xrCam.matrixWorld

  2. On capture button:

    • Create offscreen renderer (1080x2340 or 3060x4080)
    • Copy saved pose to snapCam
    • Copy xrCam.cameras[0].projectionMatrix to snapCam
    • Render scene with snapCam
    • Composite with getUserMedia takePhoto()

Result: Camera background matches perfectly, but 3D objects
are zoomed in / appear closer than in actual AR view.

Question: Is there something specific about how Needle Engine
positions the XR camera relative to scene objects that would
cause this mismatch? Is there a recommended way to do offscreen
AR capture in Needle Engine?

Thank you