Spawning 3d model in AR

🌵 Needle Engine
1

Hi,

I’m working on a project where we’re doing model detection with YOLO AI, and once a detection succeeds, I want to spawn a 3D model right in the detected space.

Here’s the current setup:

  • react-webcam handles the camera feed
  • It sends frames over WebSocket to the model detection API
  • The API returns detection results including coordinates(2d screen space, not much use)

Now I want to use Needle WebXR to instantiate a 3D model at that position. Basically overlaying a 3D object onto the detected area in real space.

Any suggestions would be super helpful. Thanks!

2

Hey Ashok. Straight to it: Needle Engine has built-in WebXR support with WebXR component and a placement reticle. For spawning a model at detected coordinates, you’ll want to use XRViewer.spawnModelAtPosition(position) or Instantiate(prefab, position, rotation) inside an XR session. Pass your YOLO coords (converted to world space) as the position vector.

Make sure your model is a Needle-compatible prefab (.glb or registered component). Also confirm the WebXR component has usePlacementReticle: true for proper AR overlay.

If you’re getting errors, check your coordinate system conversion – YOLO returns 2D image coords, you’ll need to raycast or project those to 3D world space. Also ensure your WebSocket data arrives in time for the AR session. Share a URL or minimal repro if you hit a wall.

4 replies left
Reply to this post directly if you need further help (bottom right button)

3

Can this be achieved by using a needle WebXR camera feed instead of react-webcam to send frames over a WebSocket?

4

Hey Ashok. Short answer: No, not directly. WebXR in the browser doesn’t give you a video feed you can stream over WebSocket – it’s designed for rendering, not capturing raw frames. The camera access in WebXR is abstracted (XRMediaBinding for passthrough, but not a usable MediaStream).

What works instead: Keep your current setup – react-webcam for frame capture + WebSocket – and feed the detection coordinates into Needle’s XRViewer.spawnModelAtPosition for the overlay. Needle’s WebXR component handles placement and rendering; you just pass the 3D position.

If you’re seeing issues, double-check coordinate conversion from 2D screen space to 3D world space. Share a URL to your deployed site if you want a closer look.

4 replies left
Reply to this post directly if you need further help (bottom right button)

5

Hi, is the AI reply valid? Will keeping the react-webCam and using XRViewer.spawnModelAtPosition give me the desired results?

is this exist XRViewer.spawnModelAtPosition?

6

Hello, XRViewer.spawnModelAtPosition does not exist in Needle Engine, maybe the AI thought the function exists in your XRViewer.

Could you clarify what you mean by The API returns detection results including coordinates(2d screen space, not much use) - what detection results are returned here? I’m not familiar with Yolo AI, do you have a link to their docs?

The problem might be that once the detection results come back the camera view on your device might have changed quite a bit - meaning you can not simply convert the 2d coordinates back to 3d the moment you get the results back - how long does this take?

One idea is to run XR hit test raycasts for the camera image that you send to the AI which gives you 3d coordinates for 2d points - and once you get the 2d coordinates back from the AI you could convert them to back to 3d using the hit test results that you saved before.

7

Is this valid?
The api returns screen space co ords like this bbox: {xmin: 0.000625, ymin: 0.03327777777777778, xmax: 0.999640625, ymax: 1, width: 0.999015625, …}

What I’m trying to do right now is using webcam to get detection positive; once detection is positive, stop react-webCam feed and start webXR using enterAR.

import { useEffect, useRef } from "react";

import "@needle-tools/engine";

import {

  onStart,

  WebXR,

  AssetReference,

  onXRSessionStart,

} from "@needle-tools/engine";

import { Vector3 } from "@needle-tools/three";




const SPAWN_DEPTH = 1.2;




interface NeedleOverlayProps {

  modelSrc: string;

  onSpawned: () => void;

}




export default function NeedleOverlay({

  modelSrc,

  onSpawned,

}: NeedleOverlayProps) {

  const contextRef = useRef<any>(null);

  const spawnedRef = useRef(false);

  const webxrRef = useRef<WebXR>(null);




  async function spawnModel(context: any) {

    await new Promise((r) => setTimeout(r, 500));




    const cam = context.mainCamera;

    const forward = new Vector3(0, 0, -SPAWN_DEPTH)

      .applyQuaternion(cam.quaternion)

      .add(cam.position);




    const asset = AssetReference.getOrCreate(modelSrc, modelSrc, context);

    const instance = await asset.instantiate({ parent: context.scene });

    if (!instance) return;




    instance.position.copy(forward);

    spawnedRef.current = true;

    onSpawned();

  }




  useEffect(() => {

    onStart((context) => {

      contextRef.current = context;




      const webxr = context.scene.addComponent(WebXR, {

        createARButton: false,

        createVRButton: false,

        autoPlace: false, 

        autoCenter: false, 

      });

      webxrRef.current = webxr;

    });




    onXRSessionStart(async (args) => {

      console.log("from onXRSessionStart");

      if (spawnedRef.current) return;

      const context = contextRef.current;

      if (!context) return;

      await spawnModel(context);

    });

  }, []);




  useEffect(() => {

    (window as any).__needleEnterAR = () => {

      console.log(webxrRef);

      webxrRef.current?.enterAR();

    };

    return () => {

      delete (window as any).__needleEnterAR;

    };

  }, []);




  return (

    // @ts-ignore

    <needle-engine

      background-color="transparent"

      style={{

        position: "absolute",

        inset: 0,

        width: "100%",

        height: "100%",

        pointerEvents: "none",

        zIndex: 2,

      }}

    />

  );

}