Audio Sync Advice Needed

Hi… so, I’m not sure that the following is particularly Needle and/or Unity-related but I’m hoping someone might be able to help.

Basically, I have a .glb which has a looping animation set to ‘Play Automatically’ using the animation component. This works perfectly and the animation plays exactly as expected.

I also have an audio track which needs to accompany the animation and run/loop in sync with it. However, of course even with ‘Play On Awake’ checked in the ‘Audio Source’ component the track won’t begin playing until there has been some user interaction (i.e. rotating the view of the scene or similar.)… by which time it’s out of sync with the animation which has already been running.

I know that the ‘user interaction to start audio’ thing has always existed… with all web content… since the dawn of time :upside_down_face: … but I’m trying to ascertain what might be the best solution in this instance? I don’t necessarily need the animation to start right away, so perhaps, instead of it playing automatically, there’s a way to set the animation to only begin playing on user interaction, in the same way as the audio so they run in sync with each other?

Or maybe there’s a better way to achieve what I’m aiming for?

Any thoughts or advice will be very welcome :slightly_smiling_face:

Original Post on Discord

by user 908977119781060648

:wave: so usually this is solved by having some sort of “Play” button front and center, and when someone clicks that both audio and animation play in sync

For actually playing in sync and not deviating: usually you check the time offset between audio and animation and decide which one should be corrected if the time offset becomes bigger than, say, 0.1s

We do this internally for Timeline tracks - so if you just add your animation and audio to a Timeline in Unity that will be taken care of already

As felix said: all you need to do is playing your animation via a timeline, then our engine tries to take care of that as best as possible

  1. Add a PlayableDirector component to a gameobject in your scene
  2. Create a timeline asset
  3. assign the timeline to the playable director
  4. add a audio track to the timeline and drag your audioclip onto the track
  5. add one or multiple animation tracks to the timeline and assign the animations (see docs:

You could have a look at the scrollytelling scene from our samples to see a simple reference (or import the timeline samples from unity’s timeline package via the package manager window)

Brilliant! Huge thanks, as always. That all sounds absolutely ideal - I’ll get right on it :slightly_smiling_face::+1:

by user 908977119781060648

Just an idle/curious thought… :thought_balloon: :thought_balloon: :thought_balloon: :thinking:

Given that, at the moment, the audio begins playing when a user interacts with the content without me having to add a start button or any extra controllers or timelines anything, is there a reason why I can’t ‘attach’ my animation to whatever is making that happen?

i.e. there’s obviously some sort of ‘start on user interaction’ thing in already existence, so might it be possible to somehow ‘piggy-back’ on that rather than introduce additional stuff? :thought_balloon: :thought_balloon: :thought_balloon: Or perhaps that aspect of things isn’t related to Unity or Needle?

Not a deal-breaker… I’m just musing :slightly_smiling_face:

by user 908977119781060648

You could of course hook into e.g. the (which is the threejs SpatialAudio class i think) and use the events to start and update your animation manually too and try to keep it in sync

Ooooooo, right… I’ll have a look.

It sounds a lot less elegant than your earlier suggestions, though, so probably not a great way to go. It was just curiosity, really :slightly_smiling_face:

by user 908977119781060648

You can totally do that! E.g. there could be a character with an idle animation waving at you, and once you do anything (click, drag, interact) another animation + synced audio plays

Okay, you’ve got me thinking about it more seriously, now :thinking: :slightly_smiling_face: :thought_balloon: :thought_balloon: :thought_balloon:

by user 908977119781060648

Lots to experiment with :grinning:

by user 908977119781060648

Okay, so despite several days (and nights) wrestling with this I’m really not getting anywhere with my hoped-for outcome.

It should be noted that the failings I’m experiencing are down to my own ineptitude, not Unity or Needle problems.

I’ve successfully created a PlayableDirector with an associated timeline asset - and when I set it to ‘Play On Awake’ the timeline runs as anticipated. So far, so good :grin: :partying_face:

However, what I need it to do (and what I’m singularly failing to achieve) is for it to only begin playing on user interaction.

I’ve unchecked ‘Play On Awake’ in the PlayableDirector (which successfully stops the timeline from playing automatically) but have been unable to find a way to make it start playing when required :neutral_face:

As mentioned previously, in an ideal world what I’d really like is for the timeline to be triggered to play as soon as the user begins exploring the scene (via the orbit controls, for example).

However, I haven’t had any luck trying to find general info about how to go about achieving that. So, instead, I’ve tried to trigger it via a button… but I’m obviously doing something terribly wrong as none of the various, and undoubtedly messy, permutations that I’ve concocted make it run :grimacing:

by user 908977119781060648

So, basically I’m trying to find a simple, straightforward walkthrough (if one exists) describing how to make either of the following happen:-

1). The PlayableDirector with its associated timeline starts playing as soon as there’s a user interaction (preferred).

2). The PlayableDirector with its associated timeline starts playing as soon as a button is clicked (happy to go with that if option 1 isn’t viable).

I have a horrible sinking feeling that all my troubles and woes can be placed firmly in the ‘I don’t know anything about Unity’ category. A problem compounded by the fact that I find Unity seems to insist on even the simplest of tasks being an exercise in labyrinthine torture; something which has made me reluctant to pursue it much in the past.

At nearly 60 years old… so, with time being in rather shorter supply than it was when I was younger :rofl: … I have a relatively recently acquired predilection for trying to cut straight to the chase. Sigh. I guess that’ll have to change and I’ll just have to knuckle back down and do the work :grimacing:

by user 908977119781060648

Hello @frannie :beer: can you disabling play on awake and add a tiny script that does something like this: if(this.context.input.getPointerDown(0)) ?

Hi @marcel :cactus: thanks for the suggestion… I’ll give that a go…

by user 908977119781060648

Also thank you for that detailed message! We’ll figure this out together :slightly_smiling_face:

Hmmmmmm… it doesn’t like something about my latest efforts. I’m now being repeatedly prompted to redo the export setup each time I make any changes. So, I’m going to ditch this test project and make a new version. I’ll pop back with more info when I get sorted :slightly_smiling_face:

by user 908977119781060648

Sadly, no joy thus far today.

I’ve made several new test projects (all collab sandboxes) to try the above suggestion in a fresh environment, but each time I get the following (attached)

by user 908977119781060648

by user 908977119781060648

Sorry, here’s the more comprehensive pic that I meant to post…

by user 908977119781060648