Syncing audio to animation

[brawlflower]

I'm making a rhythm game so it's important for the track animation and the audio to be as close to perfectly synced as possible, but I'm having a hard time wrapping my head around what the best way is to achieve that.

I thought I'd be clever by literally scrolling the track based on the progress of the song:

const songInstance = await resources.song.sound.play();
songInstance.on('progress', function(progress, duration) {
  track.y = progress * duration * -1000
});

This unfortunately introduced a decent amount of lag to the animation and seemingly still doesn't reliably sync the animation to the audio anyway?

I also tried relying on the ticker to animate, while positioning the track based on the progress of the song once when the audio first starts playing:

const songInstance = await resources.song.sound.play();
songInstance.once('progress', (progress, duration) => {
  track.y = progress * duration * -1000;
  app.ticker.add((delta) => {
    track.y -= delta * 16.66;
  });
}

The lag is considerably lower in this case, but the audio does not properly sync like I'd hoped it would.

Any advice or direction to be looking in the docs would be greatly appreciated!

[ZackMercury]

I'd suggest to start counting elapsed time once the song starts playing.

Please, note that deltaTime that is sent into the ticker callback has very little to do with actual time.

If you want to base your app on the actual time and you know the precise BPM of your song in order to make that work, you need to look at 

ticker.shared.elapsedMS instead of delta.

[Exca]

Usually with Webaudio you could use audiocontexts currentTime and store the starting time when song is started. From that you can calculate position = context.currentTime - soundStartedAt. This should have no delay at all.

Are you using pixi sound? Checking that source code the progress event looks like it should be correct timing. Can you see if you are using webaudio internally or htmlaudio? The later one does not have exact timing available.

[b10b]

I've also used the method described by @ZackMercury and it worked well enough to sequence events to audio reliably.  One extra thing to consider (if setting animation based on audio elapsed time) is that visuals will be tend to appear one frame behind the audio.  Because the frame is constructed and delivered after that time, whereas the audio will have continued.  This is more obvious within low framerate environments (we were doing it at 20fps, rather than 60fps for example).  A simple time offset is usually sufficient.  A related issue is to consider "un-lagging" the inputs also (that is to say, consider when they were issued, not when the input was received by the main loop).  Again this issue is less apparent at high framerates, but synchronisation should ideally be independent of fps.

[brawlflower]

Thanks everyone for your replies! I'll have to wait until after work to really dig in to all those options

21 hours ago, ZackMercury said:

I'd suggest to start counting elapsed time once the song starts playing.

Please, note that deltaTime that is sent into the ticker callback has very little to do with actual time.

If you want to base your app on the actual time and you know the precise BPM of your song in order to make that work, you need to look at 

ticker.shared.elapsedMS instead of delta.

This is a really good idea and actually how I was doing it in an earlier prototype! I'll definitely report back if this solution works for me once I've had a chance to try implementing it.

9 hours ago, Exca said:

Usually with Webaudio you could use audiocontexts currentTime and store the starting time when song is started. From that you can calculate position = context.currentTime - soundStartedAt. This should have no delay at all.

Are you using pixi sound? Checking that source code the progress event looks like it should be correct timing. Can you see if you are using webaudio internally or htmlaudio? The later one does not have exact timing available.

I installed pixi sound, but I don't know a lot about it and I actually can't tell if I'm using it at this point... Is that context available globally somehow? I'm also not sure how to find out if I'm using webaudio or htmlaudio... if it helps this is how I'm instantiating my IMediaInstance:

const songInstance = await resources.song.sound.play();

I was also previously definitely using the pixi sound library:

PIXI.sound.Sound.from({
    url: resources.song.data,
    autoPlay: true,
});

I kind of thrashed in this part of the codebase and didn't have much progress with either version just yet though.

Sorry that I sound pretty lost, I'm totally new to pixi and I'm finding the documentation for the sound library in particular pretty hard to parse for some reason (on top of this being the first project I've ever worked on where performance down to the tens of milliseconds being a concern, which is also a pretty new way of programming for me)

5 hours ago, b10b said:

I've also used the method described by @ZackMercury and it worked well enough to sequence events to audio reliably.  One extra thing to consider (if setting animation based on audio elapsed time) is that visuals will be tend to appear one frame behind the audio.  Because the frame is constructed and delivered after that time, whereas the audio will have continued.  This is more obvious within low framerate environments (we were doing it at 20fps, rather than 60fps for example).  A simple time offset is usually sufficient.  A related issue is to consider "un-lagging" the inputs also (that is to say, consider when they were issued, not when the input was received by the main loop).  Again this issue is less apparent at high framerates, but synchronisation should ideally be independent of fps.

oh, thank you! This is a great point to keep in mind. So far every time I refresh the page the timing seems a little different, so I think it'll be an achievement when I start having this problem haha

Edited January 25, 2021 by brawlflower

[brawlflower]

quick update- I made a little timing tester and found that after following some of the advice in this thread there still seems to be about 80ms of lag, with that number still being pretty inconsistent (sometimes after refreshing the lag would go up with I think the largest being around 300ms).

Some of the things I was able to figure out after experimenting with it:

- I am definitely using the pixi sound library, and it does appear that webaudio is available in the browser I've been testing in

- the context.audioContext.currentTime is globally available from the PIXI.sound.Sound object, though it doesn't look like it starts at 0 when the song starts playing, so I'm not totally sure what this signifies and I haven't been able to find documentation on it unfortunately

Unfortunately I didn't keep track of every variation that I tried, but this is my current approach:

 

const songInstance = resources.song.sound;
songInstance.play();

let millisecond = 0;
app.ticker.add(() => {
  if (songInstance.isPlaying) {
    millisecond += app.ticker.elapsedMS;
    handleHits(millisecond, pressedKeys, chart);
    track.y -= app.ticker.elapsedMS;
  }
});

I'm not happy with this solution yet, but it's still hard for me to tell where exactly the desync is being introduced, and it's honestly likely there are other factors influencing it. I'm going to take some time really diving into this for a while, but I'll definitely report back with what I learn.

Thanks for getting me in the right direction everyone!

Edited January 26, 2021 by brawlflower

[Exca]

13 hours ago, brawlflower said:

the context.audioContext.currentTime is globally available from the PIXI.sound.Sound object, though it doesn't look like it starts at 0 when the song starts playing, so I'm not totally sure what this signifies and I haven't been able to find documentation on it unfortunately

The currentTime is a value that starts increasing when you start the context. That's why the timing is done with current-start calculation. https://developer.mozilla.org/en-US/docs/Web/API/BaseAudioContext/currentTime

80ms sounds like it is not calculating correctly. Havent used pixi sound myself so not really sure what might be wrong with the progress event. If the audio is played without webaudio (which shouldnt happen on a modern browser unless explicitly told to do so) then those delays sound possible.