Dynamically Adding Image Tracking in WebXR using React - Need Help with Runtime Image Assignment

[Parsa_Amirabadi]

Hello, everyone!

I’m currently working with the Needle Engine and WebXR to implement image tracking in a React-based project. I have a component (WebXRImageTracking) that is designed to track images and attach objects or content (like a video or 3D model) to those images in an augmented reality environment.

However, I’m trying to make the image tracking dynamic and load images (and possibly video content) from a server/API at runtime, rather than having the image pre-defined in the code. My ultimate goal is to retrieve data such as the image, video, and their sizes from an API and then use this data to configure the WebXRImageTracking component during the application runtime.

Problem:

I have written some code to implement the functionality, but I’m not sure if I’m handling the dynamic addition of images to WebXRImageTracking properly. Specifically, I want to:

1. Fetch image data (like an image URL, width, and potentially a video clip) from an external server or API.
2. Dynamically load the image at runtime.
3. Set this image in the WebXRImageTracking component.

Code I’ve Written So Far:

import { Behaviour, GameObject, serializable, VideoPlayer, WebXRImageTrackingModel } from "@needle-tools/engine";
import { WebXRImageTracking, WebXRTrackedImage } from "@needle-tools/engine";
import { Object3D, Texture, Vector3 } from "three";

export class test extends Behaviour {
    @serializable(WebXRImageTracking)
    imageTracker: WebXRImageTracking | null = null;

    @serializable(WebXRTrackedImage)
    trackedSomething: WebXRTrackedImage | null = null;

    @serializable(ImageBitmap)
    targetImage!: ImageBitmap;

    @serializable(Object3D)
    targetOBJ!: Object3D;

    @serializable()
    public clip!: MediaStream;

    @serializable(Object3D)
    public videoPlayerTransform!: Object3D;

    start() {
        this.SetTargetImage(this.targetImage);
        this.SetVideoPlayer(1, 480, 720);
    }

    public SetTargetImage(image: ImageBitmap) {
        this.imageTracker = GameObject.getComponent(this.gameObject, WebXRImageTracking);
        if (this.imageTracker) {
            this.imageTracker?.trackedImages?.push(this.GetNewImageTracking(image, this.imageTracker));
            console.log(this.imageTracker?.trackedImages);
        }
    }

    public GetNewImageTracking(image: ImageBitmap, tracker: WebXRImageTracking): WebXRImageTrackingModel {
        if (this.imageTracker != null) {
            const trackedImage = new WebXRTrackedImage(
                this.imageTracker, new WebXRImageTrackingModel(), image, .15, "tracked", 0
            );
            trackedImage.model.image = "../Files/rise.jpg"; // URL or dynamically fetched image
            trackedImage.model.object = tracker!.trackedImages![0].object;
            this.targetOBJ = tracker.trackedImages![0].object?.asset as Object3D;
            trackedImage.model.hideWhenTrackingIsLost = true;
            trackedImage.model.widthInMeters = 0.25;
            trackedImage.model.imageDoesNotMove = false;
            trackedImage.model.createObjectInstance = true;
            return trackedImage.model;
        }

        return new WebXRImageTrackingModel();
    }

    public SetVideoPlayer(heightSizeInMeters: number, width: number, height: number) {
        const widthToHeightRatio = width / height;
        this.videoPlayerTransform.scale.set(
            heightSizeInMeters,
            heightSizeInMeters * widthToHeightRatio,
            1
        );
    }
}

Issue:

1. Dynamic Image and Video Loading: I’m attempting to dynamically add an image to the WebXRImageTracking component at runtime. However, I’m unsure if I’m doing this correctly when it comes to runtime image handling. The WebXRTrackedImage should be updated with new image data fetched from the API, but I’m not sure if the data assignment is happening properly in the flow of this code.

2. Runtime Behavior: The goal is to fetch image/video content, such as MediaStream, from the API, and use that as part of the WebXRImageTracking process. The content should be applied at runtime to the component without pre-defining any of the data beforehand.

What I Need Help With:

• How to dynamically fetch image and video data (along with necessary metadata like width/height) from an API and inject it into the WebXRImageTracking component at runtime.
• How to ensure the WebXRImageTracking updates properly once the new image is received, so that the 3D object or video correctly attaches to the newly tracked image.
• How to make sure the process works efficiently in a React environment, with proper handling of the WebXR API and Needle tools integration.

If anyone has experience with similar use cases or can point me in the right direction, I’d greatly appreciate your advice!

[needle-assistant (Needle Engine Assistant)]

I’m sorry, I couldn’t find a response but notified the team

[needle-assistant (Needle Engine Assistant)]

Hello Parsa Amirabadi! Based on your requirements for dynamically adding images to the WebXRImageTracking component at runtime, here’s a detailed breakdown of the steps needed to achieve that:

1. Fetch Image Data:

   • Use fetch() or a library like Axios to retrieve image data (URLs, sizes, and potential video).
   • Store the fetched data in the state.

2. Dynamically Load the Image:

   • Utilize createImageBitmap() to convert the fetched image URL into an ImageBitmap.
   • Ensure this bitmap is stored in the WebXRImageTracking component’s images map.

3. Update the WebXRImageTracking Component:

   • Modify the component to include the fetched images in its tracking list.

Here’s a snippet illustrating how you could implement this:

import { Behaviour, GameObject, VideoPlayer, WebXRImageTracking } from "@needle-tools/engine";

export class DynamicImageTracking extends Behaviour {
    @serializable(WebXRImageTracking)
    imageTracker: WebXRImageTracking | null = null;

    async fetchImageData(url) {
        const response = await fetch(url);
        const data = await response.json();
        return data; // Expecting an array of images here
    }

    async start() {
        const imageData = await this.fetchImageData('your/api/endpoint');
        for (const img of imageData) {
            const imageBitmap = await this.loadImage(img.url);
            this.imageTracker.trackedImages.push({ image: imageBitmap, widthInMeters: img.width });
            // Update tracked images in your image tracker here
        }
    }

    loadImage(url) {
        return new Promise((resolve, reject) => {
            const img = new Image();
            img.src = url;
            img.onload = async () => resolve(await createImageBitmap(img));
            img.onerror = () => reject(new Error(`Error loading image: ${url}`));
        });
    }
}

Key Points:

• fetchImageData(url): Fetches JSON data containing image details.
• loadImage(url): Loads images dynamically and converts them to ImageBitmap.
• Ensure to handle cases for server errors and handle the images properly based on the image tracker.

For more details, you can check the documentation on WebXRImageTracking.

Let me know if you need further assistance with this implementation!

1 replies left - Get a PRO license for more AI replies.
thread_BOHkm4cL3jfcqqzUIHXFj82Y
Reply to this message to continue the conversation