R   E   S   O   N   A   N   C   E 

DANIEL BREUER AND JASPER KIRTON-WINGATE

http://gitlab.doc.gold.ac.uk/jkirt001/CreativeProjects_2/tree/access

P R O J E C T   D E S C R I P T I O N

Resonance combines elements inspired by natural physics with real time audio to create an immersive audio-visual installation. The project is envisaged for use in nightclubs and other dance events. We were initially inspired by the phenomena of cymatics and synesthesia (seeing sound). As the project developed, we explored computational techniques that provide the illusion of reactive particles/materials. We realised early on that genuinely realistic simulation of cymatics is computationally impossible; this caused to look towards other creative interpretations of the concept. For example, the particle system we have used stems from autonomous agents and flocking systems, and is animated by a set of resolving forces. We have broken down our workload into two significant categories: real time audio analysis and shader generation. Resonance aims to react to audio in a aesthetically pleasing way while simultaneously being perceptively synonymous to the music. Most of the work has been done working with the Fast Fourier Transform (FFT) with real time audio input and using various shader techniques to off-load graphic processing onto the GPU which results in a smoother animation with minimal latency and ultimately a stimulating audio-visual experience. We are aiming for the finished product to have real world use in environments where visually pleasing audio interaction add to the mise-en-bande. We are also hoping to implement machine learning in partnership with the FFT in order to segment drum sounds to change the visuals depending on the percussive content of the music. We have done this so the visuals can adapt to different styles of music without the need to alter the source code.  

C R E A T I V E   P R O C E S S

 For Resonance, our creative process took a natural progression in terms of what we were able to achieve. By moving from a basic particle system to a  GLSL shader system, we were able to transform a sluggish, laggy and generally unattractive program to one that enabled us to display thousands of points on the screen that could move independently of each other and react to different sounds with relative ease. To further the project and to create a more interesting and aesthetically pleasing project, we decided to create a pulsating effect that reacted to the kick drum that would be triggered from the centre, causing the particles to spread outwards towards the edges of the screen. With some experimentation and parameter tweaking, Jasper and I came to a shared opinion that the magnitude of the pulsation of the particles should be subtle enough to let the patterns of the flocking systems settle, but also obvious enough so the audience can witness the effect of the visuals reacting to the sound.
The program was looking impressive but in order to elevate it to the next level, Jasper and I decided to have different elements of the audio input cause the program to output and react in different ways. We decided there should be multiple pulsations at different areas of the screen for each type of drum sound (kick, snare, his). This was done by specifying different frequencies from the FFT and having the respective magnitudes (converted to DB for consistency over perceptual dynamic range) react in different areas in the space, for example, the snare would cause the particles to spread from the centre whereas the bass/kick would cause the particles to spread around a force that was being delivered from the bottom of the screen, causing the entire entity to rise up and down. This gave the program an immediate improved effect, the entire project had been given another dimension as not only did the visuals react to the sound levels as a whole, but acted as a visual representation of the most important aspects of dance music; the drums. We added the pulsation from the high end of the audio not solely for aesthetic purposes but as a way to counteract an issue we faced; the kick/bass would cause the entity to shift off the top of the screen, so to solve this, we added the high end reaction and increased the magnitude of the pulsation in order to act as a barrier which stopped the program raising too high off the screen (we also used an initial offset to the central position of the forces). This would in turn serve as the foundation of the visual content of our project. The result of the particle system resonating at specific frequencies whilst continually evolving it’s dance, and the fact it is particularly reactive to percussive sounds, really had the effect we wanted in order to work in our target space of a dance/club environment..

A U D I E N C E

In order to give the project added functionality, we decided to add the ability to change the colour of the background along with the constantly changing colour of the particles themselves. This enables the user of the program to customise it to whichever colour they deem fit / appears the best to them. We also added two strobe functions (the second added after filming) which have the fascinating effect of making the project appear more three dimensional, whilst also changing the texture itself. We believe it would look extremely effective in a club environment, with the strobe being very intrusive and abrasive. After showing our friends who also enjoy going to clubs and getting positive reaction, we are sure that we have accomplished our task of creating an immersive audio visual experience designed for club and event use, we plan to test it out at the next club night which Dan helps to run - The Void.

E V A L U A T I O N

 Applying Git successfully proved to be one of the most challenging parts of the project, it is not always trivial in which order to do things, although we both recognise the importance and function of the version control system, to apply a successful model was a trial and error learning process (‘.gitignore’ caused particular issues). Despite these issues with git, which only appeared towards the final stages of the project, the delegation of roles and project in general have been very streamlined and enjoyable. We are happy with the way our code has converged and our combined efforts have created a project we are both very pleased with. Since testing began we have added numerous features such as strobing to the project, this shows that the project has the potential to adapt and expand in the future.

above image from http://www.flong.com/projects/aves/

References provided in code comments (see repo at top of blog)