SlideShare a Scribd company logo

Deep dive into Android’s audio latency problem

•
•
Sirawat Pitaksarit
Sirawat Pitaksarit

The document discusses audio latency issues on Android devices. It provides perspective on acceptable latency levels for different types of music (3-23ms) and reaction times (15-70ms). It then covers three classes of audio applications and how latency affects them differently. The bulk of the document dives into technical details of Android's audio stack, including available APIs (MediaPlayer, SoundPool, AudioTrack), hardware considerations (sampling rates, buffer sizes), and how Unity achieves low-latency audio versus a native implementation. It suggests experimenting with different buffer sizes can reduce latency but risks buffer underruns.

1 of 59
Download to read offline
Sirawat Pitaksarit Creator Exceed7 Experiments http://exceed7.com http://5argon.info
Deep dive into Android’s audio latency problem
Severity of the problem
Deep dive into Android’s audio latency problem
Deep dive into Android’s audio latency problem
In this talk, we will demystify together where exactly those time difference went to!
Some numbers, for perspective • Acceptable latency for musicians • Vocals : 3ms - voice is literally “in your head” • Drums & Percussions : 6ms - you sit right in front of them • Guitars : 12ms - may take time in the effect chains • Keyboards : 23ms - goes through several sound chambers’ acoustics Sound On Sound : Optimising The Latency Of Your PC Audio Interface
 https://www.soundonsound.com/techniques/optimising-latency-pc-audio-interface#para7
Some numbers, for perspective • Simple Reaction Time (SRT) Factors influencing the latency of simple reaction time David L. Woods, John M. Wyma, E. William Yund, Timothy J. Herron, and Bruce Reed https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4374455/
Some numbers, for perspective • Rhythm game’s highest (perfect) judgements, by player’s research • Dynamix 59ms • Deemo 50ms • Jubeat 41ms • Bang Dream! 40ms • Sound Voltex 33ms • Arcaea 25ms • Pop’n Music 25ms • Chunithm, Taiko no Tatsujin 33ms • VOEZ 30ms • Beatmania IIDX 20ms • Cytus 70ms • DDR 15ms Data compiled by @Sta_Light_
 https://twitter.com/Sta_Light_/status/1020704143231500289
15ms…is significant
Why should I care?
3 Classes of audio applications • Sequencer class - Latency does not matter • Music player : No one cares about latency. • Sequencer / Digital Audio Workstation : Able to compensate for the latency. • Instrument class - Latency may matter • Possible to “adapt to the latency”, just like a real instrument. • UI button sounds • Rhythm game class - Latency affects user experience greatly! Music is now tied to the gameplay. • Music games without response sound : compensate for latency is possible. • Music games with response sound : either perfect judgement with wrong response sound, or early judgement with in-time response sound.
Looper FL Studio Mobile Korg Gadget
Garage Band Korg iMS-20
Taiko no Tatsujin Groove Coaster Beatmania IIDX
Android Audio Primer
Each device do “prefer” something • Native Sampling Rate • Older mostly 44000Hz, newer phones moved to 48000Hz • Optimal Buffer Size • The lower the better latency-wise. Bad phone cannot handle small buffers • Low Latency Feature (true / false) • Indicates a continuous output latency of 45 ms or less • Pro Audio Feature (true / false) • Indicates a continuous round-trip latency of 20 ms or less
Each device do “prefer” something Native Sampling Rate : 44100Hz Optimal Buffer Size : 640 Low Latency : False Pro Audio : False Native Sampling Rate : 44100Hz Optimal Buffer Size : 240 Low Latency : False Pro Audio : False Native Sampling Rate : 48000Hz Optimal Buffer Size : 192 Low Latency : True Pro Audio : True Go download a test app at https://superpowered.com/latency to see yours!
Deep dive into Android’s audio latency problem
Android Audio API
Introduction to all the API options! MediaPlayer SoundPool AudioTrack (Java) AudioTrack (C++) “Tracks” OpenSL ES AAudio Thread Thread Thread HAL Java C++ Hardware Abstraction Layer
Introduction to all the API options! MediaPlayer SoundPool AudioTrack (Java) AudioTrack (C++) “Tracks” OpenSL ES AAudio Thread Thread Thread HAL Java C++ Hardware Abstraction Layer
Introduction to all the API options! MediaPlayer SoundPool AudioTrack (Java) AudioTrack (C++) “Tracks” OpenSL ES AAudio Thread Thread Thread HAL Java C++ Hardware Abstraction Layer Let’s start from the lowest level So we know what higher level APIs ultimately maps to.
AudioTrack (C++) No matter what, it must comes down to this! • Our important question : How differently Unity used the AudioTrack than our “naive” native audio playing that results in that much latency?
A short while after Unity starts up…
git clone https://android.googlesource.com/platform/frameworks/av
Track Audio Stream Mixer Thread Audio Stream
Fast Track https://source.android.com/devices/audio/latency/design#fastMixer Fast Normal
Fast Native Sampling Rate : 44100Hz Optimal Buffer Size : 240 Low Latency : False Pro Audio : False
Mixer Threads • Fast Mixer • Up to 7 fast tracks • Mix normal mixer’s submix • Track attenuation (volume) • NO : • Resampling • Effects • Normal Mixer • Up to 32 tracks • Can play any sampling rate by sample rate conversion + effects Mixer Thread Fast Mixer Thread Normal Mixer Thread Normal Mixer Thread
48000 Hz Mixer Thread 44100Hz 96000 Hz 44100Hz Sampling rate problem
Deep dive into Android’s audio latency problem
adb shell dumpsys media.audio_flinger
Example Give me 15 tracks of 48000Hz, Buffer size 196 Native Sampling Rate : 48000Hz Optimal Buffer Size : 192 Low Latency : True Pro Audio : True
Unity, your turn!
adb shell dumpsys media.audio_flinger
Phone
Your game !
How Unity achieve concurrency with just 1 AudioTrack ? • FMOD Mixer • Virtual tracks
(Normal) Mixer Thread24000 Hz FMOD Mixer (Latency!) 24000 Hz ? 44100 Hz 48000 Hz 22050 Hz (Normal) Track • Must wait until the end of frame to summarize
 `audioSource.Play()` (Latency!) • Calculate `AudioMixerGroup` buses and effects
 (Unity 5.0 +) (Latency!)
Deep dive into Android’s audio latency problem
MediaPlayer SoundPool AudioTrack (Java) AudioTrack (C++) “Tracks” OpenSL ES AAudio Thread Thread Thread Java C++ Unity startup Usually gets normal track with 24000Hz rate (Latency!) Usually gets fast track with phone’s rate
OpenSL ES • An open source set of OpenGL-style audio commands. Implementers are free to maps to anything. • It maps to `AudioTrack` (C++) on Android. • Possible to get fast track.
“Project wilhelm” git clone https://android.googlesource.com/platform/frameworks/wilhelm
AudioTrack (Java) • `new AudioTrack()` = or • Still must feed audio stream. • Possible to get fast track. • `new AudioTrack()` x32 = crash
SoundPool • `new SoundPool()` does not instantiate any track yet. • Dynamically instantiate tracks as you play. (Can set a limit) Possible to get a fast track. • Get a track index back on every play. • Supports loading a file, storing multiple audio inside, loop, pause, resume, etc.
MediaPlayer • Cannot get fast track. • Full-blown media player with video support. • Not good for latency.
Fun experiment : Crashing Unity audio
Deep dive into Android’s audio latency problem
Unity 2019.1.0a7
Now smarter! Phone Your game!
Deep dive into Android’s audio latency problem
Latency wasted on bad AudioTrack
Deep dive into Android’s audio latency problem
Buffer Size
Deep dive into Android’s audio latency problem
?
Buffer Underrun • Lower ring buffer size equals better latency, but with a risk of running out of data. • Exporting to PC build with Best Latency easily cause buffer underrun. • Exporting to Android with Best Latency usually works but I know a case that does not work.
Thank you!

More Related Content

Deep dive into Android’s audio latency problem

  • 2. Deep dive into Android’s audio latency problem
  • 3. Severity of the problem
  • 6. In this talk, we will demystify together where exactly those time difference went to!
  • 7. Some numbers, for perspective • Acceptable latency for musicians • Vocals : 3ms - voice is literally “in your head” • Drums & Percussions : 6ms - you sit right in front of them • Guitars : 12ms - may take time in the effect chains • Keyboards : 23ms - goes through several sound chambers’ acoustics Sound On Sound : Optimising The Latency Of Your PC Audio Interface
 https://www.soundonsound.com/techniques/optimising-latency-pc-audio-interface#para7
  • 8. Some numbers, for perspective • Simple Reaction Time (SRT) Factors influencing the latency of simple reaction time David L. Woods, John M. Wyma, E. William Yund, Timothy J. Herron, and Bruce Reed https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4374455/
  • 9. Some numbers, for perspective • Rhythm game’s highest (perfect) judgements, by player’s research • Dynamix 59ms • Deemo 50ms • Jubeat 41ms • Bang Dream! 40ms • Sound Voltex 33ms • Arcaea 25ms • Pop’n Music 25ms • Chunithm, Taiko no Tatsujin 33ms • VOEZ 30ms • Beatmania IIDX 20ms • Cytus 70ms • DDR 15ms Data compiled by @Sta_Light_
 https://twitter.com/Sta_Light_/status/1020704143231500289
  • 11. Why should I care?
  • 12. 3 Classes of audio applications • Sequencer class - Latency does not matter • Music player : No one cares about latency. • Sequencer / Digital Audio Workstation : Able to compensate for the latency. • Instrument class - Latency may matter • Possible to “adapt to the latency”, just like a real instrument. • UI button sounds • Rhythm game class - Latency affects user experience greatly! Music is now tied to the gameplay. • Music games without response sound : compensate for latency is possible. • Music games with response sound : either perfect judgement with wrong response sound, or early judgement with in-time response sound.
  • 15. Taiko no Tatsujin Groove Coaster Beatmania IIDX
  • 17. Each device do “prefer” something • Native Sampling Rate • Older mostly 44000Hz, newer phones moved to 48000Hz • Optimal Buffer Size • The lower the better latency-wise. Bad phone cannot handle small buffers • Low Latency Feature (true / false) • Indicates a continuous output latency of 45 ms or less • Pro Audio Feature (true / false) • Indicates a continuous round-trip latency of 20 ms or less
  • 18. Each device do “prefer” something Native Sampling Rate : 44100Hz Optimal Buffer Size : 640 Low Latency : False Pro Audio : False Native Sampling Rate : 44100Hz Optimal Buffer Size : 240 Low Latency : False Pro Audio : False Native Sampling Rate : 48000Hz Optimal Buffer Size : 192 Low Latency : True Pro Audio : True Go download a test app at https://superpowered.com/latency to see yours!
  • 21. Introduction to all the API options! MediaPlayer SoundPool AudioTrack (Java) AudioTrack (C++) “Tracks” OpenSL ES AAudio Thread Thread Thread HAL Java C++ Hardware Abstraction Layer
  • 22. Introduction to all the API options! MediaPlayer SoundPool AudioTrack (Java) AudioTrack (C++) “Tracks” OpenSL ES AAudio Thread Thread Thread HAL Java C++ Hardware Abstraction Layer
  • 23. Introduction to all the API options! MediaPlayer SoundPool AudioTrack (Java) AudioTrack (C++) “Tracks” OpenSL ES AAudio Thread Thread Thread HAL Java C++ Hardware Abstraction Layer Let’s start from the lowest level So we know what higher level APIs ultimately maps to.
  • 24. AudioTrack (C++) No matter what, it must comes down to this! • Our important question : How differently Unity used the AudioTrack than our “naive” native audio playing that results in that much latency?
  • 25. A short while after Unity starts up…
  • 29. Fast Native Sampling Rate : 44100Hz Optimal Buffer Size : 240 Low Latency : False Pro Audio : False
  • 30. Mixer Threads • Fast Mixer • Up to 7 fast tracks • Mix normal mixer’s submix • Track attenuation (volume) • NO : • Resampling • Effects • Normal Mixer • Up to 32 tracks • Can play any sampling rate by sample rate conversion + effects Mixer Thread Fast Mixer Thread Normal Mixer Thread Normal Mixer Thread
  • 31. 48000 Hz Mixer Thread 44100Hz 96000 Hz 44100Hz Sampling rate problem
  • 33. adb shell dumpsys media.audio_flinger
  • 34. Example Give me 15 tracks of 48000Hz, Buffer size 196 Native Sampling Rate : 48000Hz Optimal Buffer Size : 192 Low Latency : True Pro Audio : True
  • 36. adb shell dumpsys media.audio_flinger
  • 37. Phone
  • 39. How Unity achieve concurrency with just 1 AudioTrack ? • FMOD Mixer • Virtual tracks
  • 40. (Normal) Mixer Thread24000 Hz FMOD Mixer (Latency!) 24000 Hz ? 44100 Hz 48000 Hz 22050 Hz (Normal) Track • Must wait until the end of frame to summarize
 `audioSource.Play()` (Latency!) • Calculate `AudioMixerGroup` buses and effects
 (Unity 5.0 +) (Latency!)
  • 42. MediaPlayer SoundPool AudioTrack (Java) AudioTrack (C++) “Tracks” OpenSL ES AAudio Thread Thread Thread Java C++ Unity startup Usually gets normal track with 24000Hz rate (Latency!) Usually gets fast track with phone’s rate
  • 43. OpenSL ES • An open source set of OpenGL-style audio commands. Implementers are free to maps to anything. • It maps to `AudioTrack` (C++) on Android. • Possible to get fast track.
  • 44. “Project wilhelm” git clone https://android.googlesource.com/platform/frameworks/wilhelm
  • 45. AudioTrack (Java) • `new AudioTrack()` = or • Still must feed audio stream. • Possible to get fast track. • `new AudioTrack()` x32 = crash
  • 46. SoundPool • `new SoundPool()` does not instantiate any track yet. • Dynamically instantiate tracks as you play. (Can set a limit) Possible to get a fast track. • Get a track index back on every play. • Supports loading a file, storing multiple audio inside, loop, pause, resume, etc.
  • 47. MediaPlayer • Cannot get fast track. • Full-blown media player with video support. • Not good for latency.
  • 48. Fun experiment : Crashing Unity audio
  • 53. Latency wasted on bad AudioTrack
  • 57. ?
  • 58. Buffer Underrun • Lower ring buffer size equals better latency, but with a risk of running out of data. • Exporting to PC build with Best Latency easily cause buffer underrun. • Exporting to Android with Best Latency usually works but I know a case that does not work.