MediaPipe Unity Plugin Issue

Hello! I’m a Unity developer but new to the needle engine. I managed to setup the MediaPipe Unity Plugin (GitHub - homuler/MediaPipeUnityPlugin: Unity plugin to run MediaPipe) and run the hand tracking sample they provided.

Here’s what I tried: I followed the same structure in the sandbox sample: 1. created a Gltf object and make it the parent object for everything 2. created an export Info object, install and play

However, only buttons showed up in the browser, the web cam wasn’t on. Is there a problem with my flow? I would be very grateful for any insight.

Original Post on Discord

by user 1017896892975304704

Hi! We don’t automatically convert C# scripts and libraries as that would slow down iteration in the browser 1000x.

You can take a look at the dev branch of our samples repo for how to use the MediaPipe JavaScript library with Needle
(Clone the repo, switch to dev branch and open the project)

The relevant code for mediapipe is here:

@swang070 did this help to get you started or do you have more questions?

Thank you so much for the reply! Now I have a vague understanding of what needle engine does. Could you also explain how Needle is comparing with WebGL?

by user 1017896892975304704

Hello @swang070 here’s also some documentation about this: Our Vision 🔮 | Needle Engine Documentation

Here’s also some code in the docs: Script Examples | Needle Engine Documentation

Sorry if it’s a stupid question. I’m trying to reproduce the sample you offered on my local computer:

The project folder “needle-engine-samples/…/Mediapipe” doesn’t look like a typical Unity project so I doubt if I’m doing it correctly. I opened the folder with Unity editor and hoped to run it on the browser with a Needle export object. However, the project didn’t com with any gameobject and the only scene is “Mediapipe Hands.unity”, which I opened with errors.

Can anyone guide me how to start it? Thank you so much in advance!

by user 1017896892975304704

Hi @swang070

I think what you tried was opening a single file or folder inside a Unity package - which cant work. You install the package to a Unity project.

So here are some solutions:

  • you can either open the Unity project inside the /projects folder (in the root of the repository)

  • or install the samples package in your own Unity project by downloading the installer (see readme in the repository) and then double click that or drag-drop it into Unity’s project window https://engine.needle.tools/downloads/unity/samples

  • or you can install it in your existing Unity project if you have a project with Needle Engine already using the menu item “Needle Engine/Explore Samples”

I would recommend either solution 1 or 3

this sounds very much like checkout without lfs (or repo zip from web interface).

by user 395602247196737546

Guys, did anyone reproduced/recreated needle+mediapipe demo? How did you do it? it is not in the template list, i cannot install it to look inside to figure things out

by user 697872771497328640

@swang070 was you able to solve the issue? Can you please share :pray:

by user 697872771497328640

We’ll hopefully get a new version of the samples package published next week - what you can do in the meantime is to clone the samples repository from GitHub - needle-tools/needle-engine-samples: Live: https://engine.needle.tools/samples 🌵 Needle Engine sample scenes - use as examples and as a reference for learning 😊 and switch to the dev branch - then open the Unity project that is inside projects/Needle Engine Samples 2020_3 and then search for MediaPipe in the project view

needle-engine-samples/package/Runtime/Mediapipe at dev · needle-tools/needle-engine-samples · GitHub < this is where the media pipe stuff is. The code and package.json can be found in needle-mediapipe~ (it’s a npmdef)

Ok, I’ll try, thank you very much!

by user 697872771497328640

There is no any Unity project files inside this branch

by user 697872771497328640

It’s here: https://github.com/needle-tools/needle-engine-samples/tree/dev/projects/Needle%20Engine%20Samples%202020_3

I DID IT, Thank you so much!

by user 697872771497328640

Great!