Show custom hands model in Passthrough mode on VR Headset

🌵 Needle Engine
1

Hello my beloved needle team!
I am working on an experience, where custom hands should be visible in Passthrough mode (for example when you enter AR on Quest).

I added custom hand models for left and right hands, they are visible when I enter VR mode, but in the passthrough mode the hand models itself are not visible ( but the form of the hand models is used to occlude the camera image behind the hands). I tried one of needle-examples with custon hands (Monster-Hands) - there is the same problem, the model is not visible, but you see the mask in the form of the monster-hands (I attached an screenshot).

20241008-231602-slack
20241008-231602-slack961Ă—958 95.6 KB

I have not found way, how to turn off this “masking/occluding” and show the model itself yet. Is there a simple way to do so? What I tried to fix this was to go to XRControllerModel.ts script and comment out all lines where occlusion was applied for passthrough. But it did not changed the situation.

  if (NeedleXRSession.active?.isPassThrough) {
            model.traverseVisible((obj: Object3D) => {
               // this.makeOccluder(obj);
            })
        }

Thank you in advance :wink:

2

Hello Llllkatjallll!

To display your custom hand models in Passthrough mode while still maintaining occlusion, you need to ensure that the rendering order and visibility settings for your hand models are correctly set. Commenting out the occlusion lines might not be sufficient, as it seems the model’s visibility in Passthrough mode is governed by additional shader settings or rendering conditions.

Here’s a step-by-step guide to help you achieve this:

  1. Update Custom Hand Model Rendering: You will need to modify the hand model rendering logic to ensure the model is both visible and occludes the camera. The occlusion is generally handled separately from the model rendering.

  2. Modify the Rendering Logic: Instead of commenting out the occlusion, you can adjust the visibility settings for your custom hands directly when in Passthrough mode.

Here’s an example to illustrate this:

if (NeedleXRSession.active?.isPassThrough) {
    model.traverseVisible((obj: Object3D) => {
        obj.visible = true; // Ensure the model is visible in passthrough
        obj.material.depthWrite = true; // Enable depth writing to ensure correct occlusion
        obj.material.transparent = false; // Disable transparency if you want solid models
    });
}

  1. Ensure Model Materials are Setup Correctly: Check the materials of your hand models. Sometimes, certain shader properties might cause them to not render as expected in Passthrough mode. Ensure they are not set to any transparency settings that may hinder visibility.

  2. Debugging and Visual Feedback: Use the console to first log your model’s visibility state and material properties. This will help determine if your model is being drawn but may be obscured or incorrectly rendered.

If these adjustments do not resolve the issue, please ensure you’re using the latest version of Needle Engine, or provide further details such as the specific versions of tools and frameworks you’re using. This will help in troubleshooting more effectively.

Feel free to reach out if you have further questions or need additional clarification!

3 replies left
thread_6I1Dk52k9LY7I49tCj8kUjP1
Reply to this message to continue the conversation

3

Hi Katja, thank you for the post. We’re aware of that and will change it for custom hand models in one of the next versions so that if you use custom hands they won’t become occluders automatically

4

This topic was automatically closed 13 days after the last reply. New replies are no longer allowed.