There are a few different ways to sync visuals to music:
- Manual – live controlling visuals with keyboard or midi controls.
- Sequencing – pre-analyzing the music and scripting an animation as a list of timecoded events.
- Midi Input – if you have access to the music’s midi data, this can be a great way to drive visuals.
- Audio Reactive – code driven visuals that automatically adapt to a live audio input.
Here I want to talk about the last method. This can be useful for “Hands Free VJing”, allowing you to sit back and have the visuals automatically sync, or in a video game where you want some part of the visuals to react to the soundtrack.
The demos below work in Chrome using Three.js and the Web Audio API, but the same principals apply if you are using Processing, OpenFrameworks or some other framework.
Audio Analysis
To sync to an audio input, we need to analyse the audio stream in realtime. There are 4 main pieces of data we can extract:
- Volume – the thicker bar on the right hand side
- Waveform – the jagged white line
- Levels – the bar chart of frequency amplitudes, from bass on the left to treble on the right.
- Beat Detection – the volume bar flashes white when a beat is detected. The white line above the volume bar indicates the beat threshold.
To see what these look like, view the Audio Analysis Demo. Drag and drop an MP3 file to play it, or switch to the mic input with the control panel at right.
Volume
The volume is the current global amplitude or loudness of the track. Volume data can be used raw or eased over time to give a smoother value:
smoothedVolume += (volume - smoothedVolume) * 0.1;
Simple volume tracking can be enough to give a nice synced feel. In the Paradolia demo, the volume is used to determine the brightness of the lights in the scene. Beat detection is also used to trigger the material textures switching out.
Waveform
The waveform is the shape of the sound wave as it flies through the air and hits your ear. With the Web Audio API, use this call to get the waveform as an array of numbers between 0 and 256, where 128 indicates silence:
analyser.getByteTimeDomainData(timeByteData);
The Loop Waveform Visualizer draws the waveform data into circles that expand from the middle of the screen. The volume is also used to give a little bounce on the height of the waveform.
Levels
The levels are an array of amplitudes for each frequency range. They can be visualized as a bar chart or a 1980’s graphic equalizer. Using the WebAudio API this call will get the levels as an array of numbers between 0 to 256, where 0 indicates silence.
analyser.getByteFrequencyData(freqByteData);
In the ÜberViz demo the levels data sets the thickness of the colored strips. The smoothed volume is used to determine the size of the central white shape. The time period of the stripes movement is set to the BPM of the song. Beat detection is used to transition the camera angle. On each transition I use the Bad TV shader to do a little warping (thanks to @active_theory for the suggestion).
Beat Detection
Reliable beat detection is hard. An audio waveform is a complex shape formed by multiple sounds overlapping, so it can be hard to pick out the beat. A beat can be defined as a “brutal variation in sound energy“, meaning a beat is when the volume goes up quickly in relation to the previous value. You can do beat detection on the global volume, or by focussing on specific frequencies (e.g. to separate the bass drum from the hi-hats).
In the Audio Analysis demo we use a Simple Beat Detection Algorithm with the following logic:
- Track a threshold volume level.
- If the current volume exceeds the threshold then you have a beat. Set the new threshold to the current volume.
- Reduce the threshold over time, using the Decay Rate.
- Wait for the Hold Time before detecting for the next beat. This can help reduce false positives.
In the demo, you can play with the ‘Beat Hold’ and ‘Beat Decay’ values to try to lock onto certain beats. This type of beat detection is good for finding less frequent ‘transition points’, depending on the delay and decay values used.
Beat detection results are heavily dependent on the track you choose. To get good results you want a track with a high dynamic range (from loud to quiet) and a simple structure. I find that Dubstep in particular is hard to beat detect, since it is typically uses lots of compression (making the whole song equally loud) and has complex drum breaks.
For professional live VJing or video music production, it’s often best to combine automatic audio-reactivity with live ‘knob twiddling’ or sequencing to produce the most interesting visuals.
Happy Visualizing!
Latest Tweets
Hi Guys,
Really instructive. I also want to do something in the field of audio-reactive visuals generation.
I coded a series a test pattern in c++/openGl.
Check this one for example: http://vimeo.com/78033713
Best
Just wanted say how cool your visual example – really inspiring.
Thank you
Do you have example soft for Visual Basic 6 please.
i would love to have this as a software. im a DJ and i love visuals. this is perfect.
Vsauce3 sent me here, this is an awesome viz! any further dev plans? (can i make demands? ;-))
Again, brilliant word, thank you!
– ACE
Thank you for this amazing program! I am surprised at how genuinely marvelously this program makes mp3s into a video.
I am a video creator, and I was wondering if you could make a variant where you could upload an mp3, time the different visual effects to the detected beats, and download the result as a video file?
I experienced that sometimes when the phrase changes the visualizers change twice in one transition, the transitions’ timings are off, or they don’t change at all. Besides, I’d like to have a way to use this in a video without having to screen-cap it.
Once again, many thanks for this stunningly beautiful visualizer!
(P.S. If you’d like to take a look at my humble collection of videos, please visit my channel on YouTube.)
[…] analyze the sound and detect beats, I used this excellent article by Felix Turner as a guide. Thanks to the Web Audio API modular system, it’s easy to configure sound analysis and […]
[…] analyze the sound and detect beats, I used this excellent article by Felix Turner as a guide. Thanks to the Web Audio API modular system, it’s easy to configure sound analysis and […]
Absolute Genius! I am trying to sync wifi light bulbs to audio and this is the best resource I have found. Many thanks 🙂
Very informative. However, I have a question. I’m wanting to produce a video where the brightness of a spot of light is synced with a person’s speech.Which of these methods woukd be the simplest, most reliable and most accurate?
I’m having problems to make the mic input work 🙁
[…] で BPM 取れないかなーと試してたこともあるんですが、Felix Turner さんでさえ苦戦されているようなので、とりあえず VDMX […]
For some reason my mp3’s don’t work on the thing :/
I know this post is from 2013, but can you update the demos to use the now standardized AudioContext instead of webkitAudioContext?
[…] as well as make the experience much more immersive. My curiosity was first piqued when reading this post by Felix Turner, and then last year at Jaume Sanchez’s presentation ‘Everybody dance now with Web Audio […]
[…] Making Audio Reactive Visuals […]
For people looking into a way of controller WebGL using midi controllers i have written a simple note module that will let you select a midi input and will forward all the midi messages to a socket server, which you can connect to parse the data and control your visuals.
https://github.com/hems/midi2funk
it’s being used by my friend Hanzo when he performs visuals ( generally when i’m playing ), check his stuff out ( three.js + midi + data.gui + other little tricks )
https://www.facebook.com/h2olab
Hi guys. My friend and I have designed a new interactive lighting system that beautifully lights up and creates lovely patterns depending on the exact music beats. If you have a drop box account, you can check out this video that I have just recorded from my phone regarding the system:https://db.tt/k0DfR1Ix
My email address is kmwaurakibebe@gmail.com
[…] There is an article on Airtight Interactive that has a section about audio analysis. It gives a lot of insight into beat detection and the use of threshold to register the pulse of the music. This was helpful when I was trying to create my own beat tracking patch in max, however, as stated in the article it is very hard to get reliable beat detection, which is why i chose to use the qm.btrack instead. The article can be found here. […]
Very nice article, unfortunately, the Audio Analysis Demo is a definite No Go…
Uncaught TypeError: window.webkitAudioContext is not a constructor
at Object.init (AudioHandler.js:58)
I will say, I like the way you think. Thanks for sharing,…
Thanks for the heads up. Fixed!
Thanks for the informative tutorial the beat detecting showing it through motion graphics is one of my the best thing.
[…] https://www.airtightinteractive.com/2013/10/making-audio-reactive-visuals/ […]
Is it can do a function like this:
if waveform is low then hide image, else show image.
repeating function on when playing audio
Example:
https://i.stack.imgur.com/R3YmW.png