Record Microphone with FaceFilter

🌵 Needle Engine
1

Hi could you make a AR Mask experience on web and make a recording of yourself but also record your voice?

I know you can do all the rest but I am unsure of the microphone?

Thanks in advance!

2

Hi Justin,

To create an AR Mask experience that records both the AR interaction and your voice, you’ll need to utilize the WebXR and Voip components from Needle Engine. Here is a step-by-step outline of how to set it up:

  1. Setup WebARSessionRoot: Add this component to create the base for the AR experience.
  2. Integrate the VoIP Component: This will enable audio capturing.
  3. Create an Avatar: To visualize the AR Mask, you’ll implement an avatar system.
  4. Handle Microphone Permissions: Ensure permissions for microphone access are managed in your web application.

Here is a code snippet that integrates these functionalities:

import { WebARSessionRoot, Voip, AvatarMarker, Behaviour, GameObject } from "@needle-tools/engine";

export class MyARMaskExperience extends Behaviour {
    private arSessionRoot: WebARSessionRoot | null = null;
    private voip: Voip | null = null;
    
    awake() {
        this.arSessionRoot = GameObject.getComponent(WebARSessionRoot, this.context);
        this.voip = GameObject.findObjectOfType(Voip, this.context);
        
        if (!this.voip) {
            console.error("VoIP component is missing.");
            return;
        }
        
        // Check microphone permissions
        navigator.mediaDevices.getUserMedia({ audio: true }).then(stream => {
            console.log("Microphone access granted", stream);
        }).catch(err => {
            console.error("Microphone access denied", err);
        });
    }

    onEnterXR(args: any) {
        if (this.arSessionRoot) {
            console.log("AR Session started");
            this.arSessionRoot.onPlaced(args => {
                console.log("Scene has been placed in AR");
                this.setupAvatar(); // Call to set up your avatar.
            });
        }
    }

    private setupAvatar() {
        const avatarMarker = new AvatarMarker();
        GameObject.addComponent(this.context.scene, avatarMarker);
        // Add your specific mask/mesh here
    }
}

Key Notes:

  • Ensure that your HTML includes <script> tags for necessary libraries.
  • Make sure to handle permissions correctly for a smooth user experience.
  • You can find example setups in the Needle Engine samples repository to further study the implementation.

Feel free to adjust the avatar setup to integrate your specific AR mask. If you have any further questions or need additional components, let me know!

3 replies left
thread_RQGbGu0GNqJ8q9sx5yc2Juzl
Reply to this message to continue the conversation

3

Are you asking to record audio with the AR filters from our samples (additionally to recording the video via the menu button) ?

1 Like
4

Yes we would like to record video together with audio.
To be more specific we want to create our own 3D mask with blendshapes and switch it with this one and then we want the user to be able to record themselves with audio. (We want to change the pitch of the voice to sound more like a hamster)

image

5

Got it, sounds fun :slight_smile:

Currently the sample doesn’t support this usecase but we could prioritize it and add it in another update potentially. Please DM me your license information if you’re interested in this

1 Like
6

I see, thats nice thank you!
And is it possible to edit the sample ourselves and integrate it ourselves?

Many thanks!

7

You can copy the package/code and edit it altough you might not be able to upgrade easily anymore if we release a new version. I think most of the times it’s better to communicate what features are needed so we can expose or adapt what’s necessary for your usecase :slight_smile:

Please note that for commercial use you would still need to get a license

1 Like
8

This topic was automatically closed 13 days after the last reply. New replies are no longer allowed.