Which Needle sample is best for 3D avatar with animations

🌵 Needle Engine
1

Hello, i want to have a 3D avatar with animations that can listen to input from a microphone and then use AI API to respond to user’s voice

So, from non-technical perspective you can imagine a 3D avatar in browser that you can talk to and when it responds it looks and sounds natural because it uses ai and body-language animations.

Im mainly wondering which Needle sample will be best for getting the avatar setup with animations because i think everything else is pretty easy (the ai stuff is just an API) - and the 3D models and animations are already done (unless only a certain format of them works in web?)

Original Post on Discord

by user 352282921056468993

2

Hello there – we have an experimental AI bot :robot: that might be able to help you with your question. Would you like to try this out?

How it works:
Reply to the messages of the bot that end with a thread_id to interact with it and ask follow-up questions. You can also mention the bot directly in this channel to get help.

Help requested: please wait a moment…

3

Here is what I found for you :brain: Discord

4

Hello Shmoji,

To set up a 3D avatar with animations that can interact with users via voice input and AI in Needle Engine, you can follow these steps as mentioned in the Needle Engine documentation (see the documentation):

  1. Create a custom avatar:

    • Create an empty GameObject as the avatar root.
    • Add an Object3D named Head and add a XRFlag component set to Third Person.
    • Add Object3Ds named HandLeft and HandRight.
    • Add your 3D model and animations below these objects.

  2. Add multiplayer and networking capabilities if needed:

    • Add a SyncedRoom component for networking.
    • Add a SyncedCamera for desktop viewer sync.
    • Add a WebXRSync for XR avatar sync.
    • Add a VoIP component for voice chat.

  3. Add experimental components for avatar expressiveness (optional):

    • Use PlayerColor for a randomized synchronized color between players.
    • Use AvatarEyeLook_Rotation for eye rotations synchronized between players.
    • Use AvatarBlink_Simple for random blinking emulation.

  4. Implement body constraints:

    • Use OffsetConstraint for body following the head with leveled rotation.
    • Use BasicIKConstraint for constructing simple arm or leg chains.

Note that 3D models and animations should be in glTF or GLB format to be compatible with Web.

As for the AI and voice interaction part, it mainly involves connecting to an external AI API, which is not specific to Needle Engine but can be integrated into your custom components using web API calls.

The Needle Engine Sample Repository contains various examples that might help you get started: GitHub - needle-tools/needle-engine-samples: Live: https://engine.needle.tools/samples 🌵 Needle Engine sample scenes - use as examples and as a reference for learning 😊

If you have more specific questions or need further guidance, please let me know!

1 messages left GPT 4
thread_kSYcTw1udNOSP3VdVX1DwwVY
Thanks for your feedback!

5

feel this ai answer helped a bit, but would still like human help on what is best sample and also human take

by user 352282921056468993

6

also, if i have unity project that already does the desired functionality discussed - what all needs to be known to convert it so it works in Needle web project?

by user 352282921056468993

7

The character animation sample should be a good starting point

8

The AI answer is pretty misleading, I don’t think you need any networking for what you described

9

Like, this one and you can delete the networking parts

Synchronized Animator Synchronized Animator | Needle Engine

10

Appreciate it. If i have a Unity project already that contains an avatar you can talk to - and uses ReadyPlayerMe and InWorld Unity plugins - would i need to convert those plugins to the Javascript versions or would they just work in web without any conversion? sorry if that’s a rough question

by user 352282921056468993

11

We’re not converting C# code, typically you’d use the ReadyPlayerMe Javascript SDK directly