8 x 8

by: Maria Astakhova

8 x 8 - Safari is the most compatible browser to avoid sound popping issues

There is a lack of intuitive interfaces available through which artists interested in pursuing both sound and graphics could perform live. One imagines a software in which forms can be manipulated in a way that produces a live musical composition in real time. Eight by eight is an audio-visual program in which objects constructed from joints and springs are linked to oscillators, programmed to trigger when particles intersect. The user is free to arrange the objects as they wish within the limits of the program. Toxiclibs physics library and MaxiLib sound library are used for the physics and sound generation, respectively.  

The interface displays eight distinct vertical strings housing eight particles each along their length. Using a mouse the user may grab and pull a particle. The spring connections between neighbouring particles means that these follow the motion. As the particles travel, their collisions with particles belonging to other strings trigger sounds. Points may be glued to the sides of the screen. A string will continue wavering and intersecting with other particles for some time once the user has let go of a particle -  hence generating sound automatically for a short period. 

Sound is generated via eight oscillators, so that a maximum of eight sounds may play at the same time. When two points intersect a sound is triggered for both of them. The difference lies in the frequency of the sound, defined by the initial position of the particle along the x-axis, with the lowest x-position of a string corresponding to 100hz, and the highest to 800hz.

The important considerations were the nature of the relationship between sound and graphics, and the extent of user interaction. The spring connections between particles became an important aspect of the design early in the development process as these would define the behaviours of the particles and so facilitate a certain kind of user interaction falling somewhere between complete user control and program autonomy. Paired with the user input the springy connections would become the tool that would provide the circumstances in which movement could be created, and sound generated. For this purpose the Javascript physics library Toxiclibs is employed, allowing to connect particles via springs with a set strength and length.

The sound is developed via the Javascript sound library, MaxiLibs, and its extention library, MaxiExtras. In using the sound and physics libraries it was important to understand how the two could work together. The first stage was to build a prototype where two particles attached via a spring could generate sound. Once the logic of the libraries had become more familiar, this was not hard to achieve. Following this the project underwent several revisions as different shape structures were experimented with. It was not immediately clear whether objects ought to be in constant movement, move on and off the screen, or be static. At first the idea of constant movement was attractive as it appeared to offer more dynamism. However the aesthetic choice fell on a simple and static chain structure. Such a clean structure would lend itself easily to the discretion of the user and keep the interface clean.

Consisting of a simple chain of eight particles and their adjoining springs, the chain design has a minimalist and clinical appearance. Initially the program was envisioned to have abstract and indeterminate forms as in impressionist painting. However the stripped back aesthetic fell naturally into place and recalls the style of graphic scores by John Cage[1], Cornelius Cardew[2], and more recently SYN-Phon[3] by Candaş Şişman. Connected by sprawling lines and randomised points the graphic scores convey both moments of tension and dynamic movement – set into real movement by the spring and point arrangement of eight by eight.

With visual coherence achieved, the technial challenge entailed structuring the program in such a way that sounds could be generated by all of the particles. With sixty-four particles in total it was not possible to assign an oscillator to each one. Together with the movement of the graphics, lagging became an issue at around eight oscillators. The solution was to have eight oscillators that could be assigned to any particle instead of belonging to a fixed particle. Looping through the oscillators the program should ascertain whether one was free or not, if it was free, then the envelope triggered by a collision should be generated. This concept of a chain of logic that would seek out the availability of an oscillator was difficult to grasp as it meant that aspects of the program had to be imagined as fluid and shifting parts governed by the logic that was written inside of loops and if statements rather than fixed or hardcoded parts.

The design of the sound took on bell-like qualities, produced via frequency modulation – this was suited to the light and agile appearance of the strings. Onscreen rotary knobs on the left-hand side give user control over the frequency and amplitude of the modulating frequency. The modulating frequency acts upon the carrier frequency (defined by the initial x-position) of a particle. By mapping the frequency of the sound to the initial x-position of each string, eight different notes became possible. With the radius of the particle linked to the volume of the sound it is making, the user can see an immediate visual return on their actions within the program. Thus a spatial relationship between the sound and the layout of the graphics is set up. The various metallic and bell-like sounds that result from different configurations of these parameters recall the work of John Chowning who first discovered FM synthesis. Particularly Chowning's 1977 compisition Stria[4] in which bell and drone-like sounds slip between meditative wanderings and brooding discord. 

An important step between static graphical scores and interactivivity is Simon Katan’s Cube with Magic Ribbons[5]. A live performance audio-visial programme in which a shifting environment - akin to a graphic score - is controlled by the user. The project demonstrates the possible relationships that can be developed between sound and graphics, as well as the live-performance capabilities of audio-visual software. The shifting environment and clear aural returns on the visual changes effected by the user combine to make an engaging and convincing platform for audiovisual performance. Notably, through its shifting landscape, the program also offers sequencing capabilities. The work offers important critical perspective from which to consider the developments of softwares that aspire to audio-visual performance such as eight by eight.

Inside of eight by eight there are obvious limitations in terms of building up a sequence or musical narrative. While a some amtosphere can be built up through arranging the strings and altering the modulation values, the sound is static. In this sense the program is more akin to an installation rather than a musical performance where the music would have a clear progression.

However even within an installation setup there is scope for more dynamic behaviours. The challenge was to design a program in which sounds and graphics influenced each other. It would thus be good to develop the work so that the sound values had a greater impact on the the visual set up, for example the thickness, strength and length of the springs connecting the particles could be subtly or dramatically changed by some of the changing values in the sound parameters. This visual movement could then feedback into the sound values in order to produce a dynamically changing sonic and graphic landscape. If the sounds developed as a result of the interactions between the objects, this would negate the need for rotary knobs and create a more unpredictable experience for the user. The consideration here is how much autonomy this would leave for a user who wants more control over the performance.

Within the gallery space the project consisted of a screen and mouse set-up. Generally visitors' interests were piqued sufficiently to put on the headphones. Users found the interface intuitive enough to work out the demands of the program, learning the nature of the sound and graphic connection. The availability to alter given parameters via the onscreen rotary knobs extended the time which users experimented with the sound qualities. It was found that users had a tendency to seek out a sound that they preferred and spend more time playing the program as an instrument with that sound. Users responded favourably to the project, comments were made on the possibility of the sound developing as a result of their interactions with the strings rather than manually adjusting parameters via the rotary knobs, and a criticism was made on the lack of sequencing possibilities.

Throughout the project the most difficult task to navigate was the solving of technical and structural issues in tandom with the development of the finalised version of the project. The experience was often a misalignment between artistic vision, technical constraints, and technical knowledge. The reality was that these things developed in reaction to, and as a result of each other. In its final form the clinical and clean interface of the program resembles a virtual laboratory for audio experimentation rather than an audio-visual spectacle. Both of these avenues are interesting ones to pursue, and suggest different ways in sound and graphics could interact as well as the degree of user control.

Source Code:

http://gitlab.doc.gold.ac.uk/masta002/8x8

References

[1] John Cage, Fontana Mix (1958) - http://www.medienkunstnetz.de/works/fontana-mix/

[2] Cornelius Cardew, Treatise (1967) - https://www.youtube.com/watch?v=JMzIXxlwuCs

[3] Candaş Şişman, SYN-Phon (2014) - http://www.csismn.com/SYN-Phon

[4] John Chowning, Stria (1977) -  https://www.youtube.com/watch?v=988jPjs1gao

[5] Simon Katan, Cube With Magic Ribbons (2012) - https://vimeo.com/36888504

Javascript Libraries:

Toxiclibs - haptic-data.com/toxiclibsjs

MaxiLib - no public access. Javascript DSP library built at Goldsmiths

MaxiExtras - no public access. Javascript DSP library built at Goldsmiths