Rendering stereoscopic side-by-side image

I’ve tried this dozens of different ways. In my XRCameraController.ts script, layer changes to main camera affects all eyes, even when in VR mode. Any layer changes to WebXRArrayCamera or any WebXRCamera in the array have no effect. I’ve tried various layers to avoid conflicts such as left eye layer 8 and right eye layer 9.

Layer changes on meshes in scene works.

I created a simple three.js page that runs on the same webserver using the same three.js library in the same folder as the needle app. Changes to layers works on its WebXRArrayCamera yielding the desired stereoscopic effect.

I’ve attached everything here.

Zip file of indexvideo.html, images, css that works on vanilla three.js:

XRCameraController.ts (for cameras), SetLayer.ts (for meshes) zipped:

Let me know if you want a copy of my whole Unity project.

Thank you in advance!
Darius

Hi Darius,

Thanks for providing details about the issue with your WebXR setup. It seems like you’re facing issues with layer management for stereoscopic rendering using the WebXRArrayCamera.

To address this, here are some steps you can try:

  1. Ensure Camera and Layer Setup: Verify that your WebXRArrayCamera is correctly set up and that both left and right eye cameras are configured to respect different layers. In your XRCameraController.ts, ensure you’re manipulating the correct cameras corresponding to the right layers.

  2. Layer Assignment: Make sure that you’re not only setting the layer on the camera but also on the meshes in the scene that you want to render for that specific camera. This might include checking the layer settings in the SetLayer.ts script.

  3. Debugging: Log the layers of both the WebXRArrayCamera and your scene objects to ensure they’re being set properly. Sometimes knowing the state of the layers at runtime can give insights into what’s going wrong.

  4. Experiment with Layer Masks: When you set layers for the WebXRArrayCamera, make sure you also adjust the layer masks correctly if your objects are still rendering on the wrong eye. This can often mean setting up masks for the left and right cameras distinctly, depending on their intended visuals.

  5. Different Camera Instances: If both eyes are rendering the same content, check if you are working with separate instances of the WebXRCamera. Each instance should have distinct settings that you configure individually.

If you continue to experience issues, you might consider sharing the exact changes made in your code for deeper inspection or potentially a zip of the whole Unity project if the problem is still elusive.

If you provide your specific XRCameraController.ts and SetLayer.ts files with the relevant portions that handle camera and layers, I can help further!

Feel free to ask if you have additional questions.

1 replies left - Get a PRO license for more AI replies.
thread_xjCV62Cc4n9nh5qDtyD9b9wg
Reply to this message to continue the conversation

// src/scripts/XRCameraController.ts
import { Behaviour, serializable, Context } from “@needle-tools/engine”;
import * as THREE from “three”;

// Define your layers (consider putting these in a shared constants file)
const LEFT_EYE_LAYER = 8;
const RIGHT_EYE_LAYER = 9;
const DEFAULT_LAYER = 0;
// Add any other layers you might use
const UI_LAYER = 5;

export class XRCameraController extends Behaviour {
@serializable()
targetLayer: number = DEFAULT_LAYER;

// Keep track if XR layers were set to avoid redundant logs/operations
private xrLayersSet: boolean = false;

start() {
        this.setCameraLayers();
}

// Optional: Use update or lateUpdate if layers need to change dynamically
// lateUpdate() {
//    // Example: Change layer based on some condition
//    // if (someCondition) {
//    //     this.setCameraLayers(NEW_LAYER);
//    // }
// }

/**
 * Sets the layers for the specified cameras (Main and/or XR).
 * Call this method from other scripts or animations to change layers dynamically.
 * @param newTargetLayer The primary layer the camera should render.
 */
public setCameraLayers() {
    this.xrLayersSet = false; // Reset XR flag if called again

    console.log(`Setting camera layers to target: ${this.targetLayer}, include default: ${this.targetLayer}`);

    // --- Apply to Main Camera ---
    //this.configureCameraLayers(this.context.mainCamera, DEFAULT_LAYER, "Main Camera");
    this.context.mainCamera.layers.disableAll();
    this.context.mainCamera.layers.enable(this.targetLayer);
    this.context.mainCamera.layers.enable(LEFT_EYE_LAYER);
    this.context.mainCamera.layers.enable(RIGHT_EYE_LAYER);

     // --- Apply to XR Cameras ---
    // We often need to do this continuously in update/lateUpdate for XR,
    // as the XR manager might reset layers. The XRCameraController example
    // does this in lateUpdate. If using *this* script *instead* of
    // XRCameraController for layers, you might move the XR part to lateUpdate.
    // For simplicity here, we'll do it once when called.
    this.trySetXRLayers();

}

// Use lateUpdate to potentially override XR settings if needed, similar to XRCameraController
lateUpdate() {
    // If applying to XR, ensure layers are set correctly each frame,
    // especially if another system might be interfering.
    if (this.context.renderer?.xr?.isPresenting && !this.xrLayersSet) {
         this.trySetXRLayers();
    }
     // Reset flag if no longer presenting
    if (!this.context.renderer?.xr?.isPresenting) {
        this.xrLayersSet = false;
    }
}


private trySetXRLayers() {
    const renderer = this.context.renderer;
    if (!renderer) return;
    const xrManager = renderer.xr;

    // Check if XR is active and we have the camera array
    if (xrManager && xrManager.enabled && xrManager.isPresenting) {
        const xrCamera = xrManager.getCamera() as THREE.WebXRArrayCamera;

        if (xrCamera && xrCamera.isArrayCamera && xrCamera.cameras.length >= 2) {
            // Configure Left Eye (typically uses LEFT_EYE_LAYER or the main targetLayer)
            // Decide if XR eyes should see *only* their specific layer + default,
            // or if they should see the general 'targetLayer' + default.
            // Let's assume they should see their specific eye layer + default for stereo separation.
            this.configureCameraLayers(xrCamera.cameras[0], LEFT_EYE_LAYER, "XR Left Eye");

            // Configure Right Eye
            this.configureCameraLayers(xrCamera.cameras[1], RIGHT_EYE_LAYER, "XR Right Eye");

            this.xrLayersSet = true; // Mark as set for this frame/session
            xrCamera.layers.enable(this.targetLayer);
            xrCamera.layers.enable(LEFT_EYE_LAYER);
            xrCamera.layers.enable(RIGHT_EYE_LAYER);
        }
        // Optional: Log if XR is presenting but camera isn't ready yet
        else if (this.context.time.frameCount % 120 === 0) { // Log less frequently
             console.log("CameraLayerController: XR Presenting, but ArrayCamera not valid yet.");
        }
    }
}

/**
 * Helper function to configure layers for a single camera.
 */
private configureCameraLayers(camera: THREE.Camera, layerToEnable: number, cameraName: string) {
    // Start fresh by disabling all layers
    camera.layers.disableAll();

     // Enable the target layer
    camera.layers.enable(layerToEnable);

    // Log the resulting mask for debugging
    console.log(`${cameraName} layers set. Mask: ${camera.layers.mask}`);
}

}

// Updated SetLayer.ts
import { Behaviour, serializable } from “@needle-tools/engine”;
import * as THREE from “three”;

const LEFT_EYE_LAYER = 8;
const RIGHT_EYE_LAYER = 9;
const DEFAULT_LAYER = 0;

export class SetLayer extends Behaviour {
@serializable()
myLayer: number = DEFAULT_LAYER;

start() {
    const obj3d = this.gameObject as THREE.Object3D;

    if (obj3d) {
        // Apply layer recursively to this object and all its children
        obj3d.traverse((child) => {
            // Check if it's a Mesh or potentially other renderable types if needed
            // if (child instanceof THREE.Mesh /* || child instanceof THREE.Line etc. */) {
                child.layers.set(this.myLayer);
            // }
        });

        // Log the mask of the root object this script is attached to for confirmation
        console.log(`${this.gameObject.name} root and children layers set for layer ${this.myLayer}. Root mask: ${obj3d.layers.mask}`);

        // Optional: Log mask of first Mesh child found for deeper debugging
        const meshChild = obj3d.getObjectByProperty("isMesh", true);
        if(meshChild) {
            console.log(`  - First mesh child (${meshChild.name || 'unnamed'}) mask: ${meshChild.layers.mask}`);
        }

    } else {
        console.warn("Could not find Object3D for layer assignment on", this.gameObject.name);
    }
}

}

Hi, that’s default webxr behaviour in threejs.

The webxr camera layers are untouched by Needle Engine and the WebXRManager is a threejs class that handles stereo rendering - three.js synchronizes the camera layers

See the code here in threejs:

Have you tried to check that?

A workaround might be to patch the camera render function and modify render layers in there for the xr camera (or another function to override this behaviour).

Please dont open Multiple posts for the same question: How to render a stereoscopic side-by-side image in a scene for VR viewing? - #7 by socinian

Thanks. I’ll look at that three.js code and make sure it gets called.

I wasn’t sure how to add the “Bug” tag, which is why I created the second post. I’ll be more concise in the future.

Hi, how do you mean be sure it gets called? This code runs every frame - or do you mean you look into how to override it?

If I understand the three.js code right, it’s hardcoded that layer 1 is always shown on the left eye and layer 2 is always shown on the right eye. Thus, if you want to have objects show on specific eyes, your camera should neither render layer 1 nor 2 by default – then they will be visible in the respective eye.

Out of time to try that for today – maybe you can test it out!

Thanks! I’ll try setting the camera to neither render layer 1 nor 2 by default and then set or enable the respective eye layer.

Interestingly, my version of three.js in my build folder is v.169 which doesn’t have those lines of code in WebXRManager.js that Marcel referenced which seems to be in v.176. I’m trying npm install three@latest in the build folder, but exporting to local browser seems to overwrite it.

They do but the code is written differently: three.js/src/renderers/webxr/WebXRManager.js at 78a847407afe0cbec4ee7ffa1fad8a19a8013ffe · needle-tools/three.js · GitHub

Sorry should have sent this link in the first reply.

OK. I’ve simplified my testing to the point that I can’t simplify it any more with my current understanding of WebXR in three.js. But the demo should be simple enough for someone with experience to understand it. I’m now to the point that any more simplification breaks it. My debugging is going in circles.

Here’s an example of my problem:

It uses one Typescript script added to one empty gameObject in the default Needle engine sample scene. If one clicks on the Needle provided “Start VR” in the Quest then one has use of the VR rig & controllers, but the stereoscopic images are rendered in 2D. If one, instead, clicks the “Start VR” button I created and added to the scene (slightly behind and below the Needle one) the stereoscopic images render correctly, but the user no longer has rig VR controllers. Pointing me in the correct direction would be appreciated, even if its links to docs. Here’s my zipped project. It’s a default 3D renderer in Unity 6 (6000.0.25f1) with just Needle Engine package added and the default Needle template scene with my script and 3 images added.

My StereoscopicViewer.ts file:

Hi Darius,

If I understand the script shared correctly then you try to setup the XR layers immediately at the start() event in your component. Assuming your script exists in the scene right from the beginning then there is no XR camera yet.

You probably want to use the onEnterXR callback

export class StereoscopicViewer extends Behaviour {

    onEnterXR(_args: NeedleXREventArgs): void {
        const camera = this.context.xrCamera;
        console.log("Camera", camera);
    }
}

Thank you! I’ll try that next.

I do however think the original “problem” persists - which is that three’s WebXRManager syncs the layers of the XRCamera with the layers of the camera used for rendering

I’ve added the onEnterXR callback. It gets called in my code and I try to set all the camera layers at that time. However, no change in behavior. Using your button shows a non-stereoscopic version of the loaded images when in XR mode on the Meta Quest 3. Clicking the button I added programmatically that calls VRButton.createButton(this.context.renderer) renders the three loaded images stereoscopically.

    // Add VR button
    document.body.appendChild(VRButton.createButton(this.context.renderer));

} // end of init()

I really don’t want to take my time away from content creation to patch WebXRManager. And I find it odd that a different button makes it work fine w/o patching WebXRManager.

This demonstrates the difference in a running webpage to be viewed in Meta Quest headset: