MediaPipe Unity Plugin Issue

discord-import · July 3, 2023, 12:00am

Hello! I’m a Unity developer but new to the needle engine. I managed to setup the MediaPipe Unity Plugin (GitHub - homuler/MediaPipeUnityPlugin: Unity plugin to run MediaPipe) and run the hand tracking sample they provided.

Here’s what I tried: I followed the same structure in the sandbox sample: 1. created a Gltf object and make it the parent object for everything 2. created an export Info object, install and play

However, only buttons showed up in the browser, the web cam wasn’t on. Is there a problem with my flow? I would be very grateful for any insight.

Original Post on Discord

by user 1017896892975304704

hybridherbst · July 3, 2023, 12:00am

Hi! We don’t automatically convert C# scripts and libraries as that would slow down iteration in the browser 1000x.

You can take a look at the dev branch of our samples repo for how to use the MediaPipe JavaScript library with Needle
(Clone the repo, switch to dev branch and open the project)

hybridherbst · July 3, 2023, 12:00am

The relevant code for mediapipe is here:

github.com

needle-tools/needle-engine-samples/blob/dev/package/Runtime/Mediapipe/needle-mediapipe~/MediapipeHands.ts

// START MARKER using mediapipe with hands
import { FilesetResolver, HandLandmarker, HandLandmarkerResult, NormalizedLandmark } from "@mediapipe/tasks-vision";
import { Behaviour, Mathf, serializable, showBalloonMessage } from "@needle-tools/engine";
import { ParticleSphere } from "./ParticleSphere";

export class MediapipeHands extends Behaviour {

    @serializable(ParticleSphere)
    spheres: ParticleSphere[] = [];

    private _video!: HTMLVideoElement;
    private _handLandmarker!: HandLandmarker;

    async awake() {
        showBalloonMessage("Initializing mediapipe...")

        const vision = await FilesetResolver.forVisionTasks(
            // path/to/wasm/root
            "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm"
        );

This file has been truncated. show original

hybridherbst · July 4, 2023, 12:00am

@swang070 did this help to get you started or do you have more questions?

discord-import · July 4, 2023, 12:00am

Thank you so much for the reply! Now I have a vague understanding of what needle engine does. Could you also explain how Needle is comparing with WebGL?

by user 1017896892975304704

marcel · July 4, 2023, 12:00am

Hello @swang070 here’s also some documentation about this: Our Vision 🔮 | Needle Engine Documentation

marcel · July 10, 2023, 12:00am

Here’s also some code in the docs: Script Examples | Needle Engine Documentation

discord-import · July 18, 2023, 12:00am

Sorry if it’s a stupid question. I’m trying to reproduce the sample you offered on my local computer:

github.com

needle-tools/needle-engine-samples/blob/docs/code-marker/package/Runtime/Mediapipe/needle-mediapipe~/MediapipeHands.ts

// START MARKER using mediapipe with hands
import { FilesetResolver, HandLandmarker, HandLandmarkerResult, NormalizedLandmark } from "@mediapipe/tasks-vision";
import { Behaviour, Mathf, serializable, showBalloonMessage } from "@needle-tools/engine";
import { ParticleSphere } from "./ParticleSphere";

export class MediapipeHands extends Behaviour {

    @serializable(ParticleSphere)
    spheres: ParticleSphere[] = [];

    private _video!: HTMLVideoElement;
    private _handLandmarker!: HandLandmarker;

    async awake() {
        showBalloonMessage("Initializing mediapipe...")

        const vision = await FilesetResolver.forVisionTasks(
            // path/to/wasm/root
            "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm"
        );

This file has been truncated. show original

The project folder “needle-engine-samples/…/Mediapipe” doesn’t look like a typical Unity project so I doubt if I’m doing it correctly. I opened the folder with Unity editor and hoped to run it on the browser with a Needle export object. However, the project didn’t com with any gameobject and the only scene is “Mediapipe Hands.unity”, which I opened with errors.

Can anyone guide me how to start it? Thank you so much in advance!

by user 1017896892975304704

marcel · July 18, 2023, 12:00am

Hi @swang070

I think what you tried was opening a single file or folder inside a Unity package - which cant work. You install the package to a Unity project.

So here are some solutions:

you can either open the Unity project inside the /projects folder (in the root of the repository)
or install the samples package in your own Unity project by downloading the installer (see readme in the repository) and then double click that or drag-drop it into Unity’s project window https://engine.needle.tools/downloads/unity/samples
or you can install it in your existing Unity project if you have a project with Needle Engine already using the menu item “Needle Engine/Explore Samples”

marcel · July 18, 2023, 12:00am

I would recommend either solution 1 or 3

discord-import · July 18, 2023, 12:00am

this sounds very much like checkout without lfs (or repo zip from web interface).

by user 395602247196737546

discord-import · July 21, 2023, 12:00am

Guys, did anyone reproduced/recreated needle+mediapipe demo? How did you do it? it is not in the template list, i cannot install it to look inside to figure things out

by user 697872771497328640

discord-import · July 21, 2023, 12:00am

@swang070 was you able to solve the issue? Can you please share

by user 697872771497328640

marcel · July 21, 2023, 12:00am

We’ll hopefully get a new version of the samples package published next week - what you can do in the meantime is to clone the samples repository from GitHub - needle-tools/needle-engine-samples: Live: https://engine.needle.tools/samples 🌵 Needle Engine sample scenes - use as examples and as a reference for learning 😊 and switch to the dev branch - then open the Unity project that is inside projects/Needle Engine Samples 2020_3 and then search for MediaPipe in the project view

marcel · July 21, 2023, 12:00am

needle-engine-samples/package/Runtime/Mediapipe at dev · needle-tools/needle-engine-samples · GitHub < this is where the media pipe stuff is. The code and package.json can be found in needle-mediapipe~ (it’s a npmdef)

discord-import · July 21, 2023, 12:00am

Ok, I’ll try, thank you very much!

by user 697872771497328640

discord-import · July 21, 2023, 12:00am

There is no any Unity project files inside this branch

by user 697872771497328640

marcel · July 21, 2023, 12:00am

It’s here: https://github.com/needle-tools/needle-engine-samples/tree/dev/projects/Needle%20Engine%20Samples%202020_3

discord-import · July 21, 2023, 12:00am

I DID IT, Thank you so much!

by user 697872771497328640

marcel · July 21, 2023, 12:00am

Great!

Topic		Replies	Views
Device camera not rendering with 3d background 🌵 Needle Engine question	7	13	July 12, 2023
VFX from Unity to Three.js 🌵 Needle Engine question	54	122	July 14, 2023
Needle.Tools Source Code Request 🌵 Needle Engine question	11	45	October 10, 2023
Scene Loading issues with needle 🌵 Needle Engine question	101	139	January 11, 2024
Spatial Audio Demo Error 🌵 Needle Engine question	9	33	January 18, 2024

MediaPipe Unity Plugin Issue

Related topics