The Persistence Of Sound
Becky Johnson and Itonisha Rowe
http://gitlab.doc.gold.ac.uk/ipatt001/CP
‘The Persistence of Sound’ is an environment that the user can explore using the TouchOSC control interface. It is also built using openFrameworks and the Maximillian sound library. This art game plays with the concepts explored within the Surrealist[1] art movement and pulls reference from the movements muse, Sigmund Freud and his thesis ‘The Interpretation of Dreams’[2]. The project also relies upon the relationship between the audio and visual elements to create a one-on-one immersive experience.
Intended Audience and Outcome
The concept of art house games is thought to have originated from the early Surrealist collaborative drawing game, ‘Exquisite Corpse’[3]. For this reason, we felt that a contemporary art game would be a pertinent structure for the exploration of the concepts found within Surrealism. Whether it be an art house film or game, the intended audience normally covers a large age range. However, these people will usually be both highly passionate about the execution of experimental visuals and analytical about its underlying contextual reference.
‘The Persistence of Sound’ is a work of creative expression that concerns itself mainly with the generation of mood and atmosphere; two integral elements that define art house works. The title itself is derived from Salvador Dali's 'The Persistence of Memory'[4] to establish a link between the project and the Surrealist movement for the audience.
The project takes simple navigation and a projected display to help generate an engaging experience. It aims to subtly break the laws of logic whilst retaining a simple earthly allure. This is in attempt to pay homage to the surrealist movement and the ongoing discussion of dreams and the subconscious. This interactive experience targets its audience through the careful design, selection and implementation of the environment and its soundscape.
Our intended outcome was to create a game in which the user can become totally consumed in; a world that resonates with our own reality and provokes thought about conscious boundaries. ‘The Persistence of Sound’ offers a chance of momentary nonchalant escapism which intends to be purely for fun. It has been designed to require minimal efforts to operate hence the simple navigation and the absence of a point system. Along with this, it’s illustrative references that draw upon the discussions of the subconscious allows for the gamer to use ‘The Persistence of Sound’ as a virtual space to reflect and think.
Overall, we wanted to venture into the content of dreams and surrealism in a light-hearted manner and hope that the user can extrapolate their own interpretation and purpose for the piece.
Background Research
Surrealism sought to revolutionise the general perception of the rational mind. The rules of society were seen as being oppressive and the exploration of the subconscious was a quintessential element to alleviate this constraint as it was thought that the subconscious tells the truth. For our background research we gathered both aesthetic and theoretical inspiration through the study of relevant artwork, games, and literature. We wanted to gain an insight as to the way in which Surrealists conveyed these ideas through the use of form, colour, composition and subject matter.
Surrealists often use automatism which involves the suppression of conscious control over the art making process, acting spontaneously and intuitively. Fusing this free fluid style with hyper realism meant that objects were often distorted. The choice of composition normally results in objects being isolated or being set to abstruse angles so that the observer can no longer distinguish between dream and reality. The brain is thought to index different stimuli, continuously referring back to the memory file in order to find connections and fill in the blanks. Along with this, there is usually a bizarre juxtaposition present, placing seemingly unrelated elements together. This is in attempt to coerce their observers to make new connections and see things in a different light. These elements are usually brought together using a cool saturated colour palette in pursuance of seducing the observer into a dream-like state.
“The dream shows how recollections of one’s everyday life can be worked into a structure where one person can be substituted for another, where unacknowledged feelings like envy and guilt can find expression, where ideas can be linked by verbal similarities, and where the laws of logic can be suspended.”
~Sigmund Freud, The Interpretation of Dreams
The contemporary British surrealist artist, Glenn Brown[5], uses a piled and smeared effect when applying oil paints. We have tried to incorporate this when texturing our objects in ‘The Persistence of Sound’ in an attempt to reference Brown’s painting technique. We have also adopted Salvador Dali and Gyuri Lohmuller’s[6] use of composition and placement of objects, as we found them to be very effective when translated into a virtual space. Furthermore, we wanted to regenerate the subdued but slightly anxious mood and atmosphere that Yves Tanguy’s[7] gives off in his work through his use of light. The consideration that was needed to incorporate these elements into the design of our environment whilst making an original piece of work has been a delicate process.
Some examples of Glenn Brown's use of texture are shown below.
Creative Process
“There is no exquisite beauty… without some strangeness in the proportion.”
~ Edgar Allan Poe
Our creative process involved generating colour palettes from surrealist pieces. This was to further develop our understanding of the use of colour in artwork to be able to then model our environment's colour scheme in a similar manner.
Our experiments showed us that the palettes include bold and natural colours and so we chose to use colours of an earthly palette for our landscape. We decided also that we wanted to include a feature of an artificial nature that contrasted our chosen colour palette. As a result, we chose to create a checkerboard flooring as we felt that the strong monochromatic colours would compliment the colour palettes that we studied. From our earlier background research, we felt that this pattern is a well-known Surrealist trademark and would provide a link between our project and existing pieces that the audience would recognise.
We then experimented with the sky box, using different palettes to determine what would achieve the mood and atmosphere that we aspired for. We decided to keep the natural blue and white hues for the sky to keep in touch with the element of reality that is present within surrealistic art. Within surrealism the atmosphere is usually set with an overall mood through the use of composition, subject matter, colour, texture, and form as a collective. They often refrain from changing the natural attributes of their works in terms of the landscape or setting, only skewing the images actual subject themselves.
In our creative process, we deliberated about what form the environment as a whole would take. In the end, we chose to create a floating island which would stand as the foundation for the landscape to be built upon. This choice was developed from both technical and aesthetic reasoning. Technically, this would eliminate the problem of the user being able to reach the horizon that the sky box produced. Aesthetically, we believed that this feature would express the sense of isolation that is often associated with surrealist art.
Another aspect of the creative process involved researching into procedural textures through the use of shaders. We were inspired by the selection of procedural texture examples shown in ‘The Book of Shaders'[8] by Patricio Gonzalez Vivo and Jen Lowe. The images below in particular influenced our decision to create a shader to implement a water-like texture.
Our choices in models stemmed from our research into the content of dreams and the subconscious. As the movement of Surrealism and the content of the subconscious mind are linked so heavily, we felt that basing our model choices around this was an interesting idea. Our choice of models included teeth as they are a prevalent subject in dreams and are thought to connote powerlessness, vanity, and health. Another theory is that they are a reflection of your anxieties and how others may perceive you. We also decided to include models of TVs as upon research they are stated to represent your mind and its flowing thoughts. It is said that the dream reflects how you are receiving, integrating, and expressing your ideas and thoughts and that the programs you dream of watching are an objective view of the things that are in your mind.
The audio was another element of ‘The Persistence of Sound’ that required carefully planning and production. Our intention was to produce a soundscape that was simultanously contemporary and surreal that reflected the visuals.
We researched into existing surrealist music, and in particular a piece called the 'Butterfly Landscape'[9] by Another Perspective provided us with a lot of inspiration. This piece is based upon Salvador Dali's piece, 'Butterfly Landscape (The Great Masturbator In A Surrealist Landscape With D.N.A)'[10] and attempts to create a living surrealistic environment through the soundscape.
This research gave us the aim to use the technique of automatism for the production of the soundscape as we wanted the process to be organic and impulsive and for the result to reflect this. With help from another student in the Computing department, Joshua Hodge, we used Ableton to take a mixture of preset samples and recorded midi snippets to build up a track. More specifically, we added an upbeat electronic drum rhythm behind the percussion and synthesised keys. We felt that these audio components as a collective suggested a slight feeling of suspense, and heightened the sense of isolation within a territory of the unknown.
Commentary on Build
Our process began by experimenting with the TouchOSC interface for the control of navigation using an iPad. We wanted to create a easy-to-use interface with minimal buttons so we decided that the use of the accelerometer to control the user's rotation would be effective. Initially, we had the idea that we would have two interfaces, one for the navigation as mentioned and another for the user to select from preset music samples. This would have allowed the users to build a track that acted as the soundscape which would have then affected the way in which the environment would react and transform through the use of FFT. We decided against this for two reasons. Firstly, we realised that this would require a stand-alone TouchOSC interface to control the sample selection. We felt that this would be asking too much of the user to control both the navigation and the samples and in turn would take there attention away from the visuality of the project. Secondly, each model in the environment would have to be programmed to react individually to each sample and this in turn would become very performance-expensive. As a solution, we decided to produce one soundscape so that we would only have to perform FFT on this sample.
We then focused on the generation of the terrain and experimented with different methods using openFrameworks to decide which algorithm we wanted to use. This involved small tests using the Diamond-square algorithm and noise. We concluded that the use of Perlin noise to create the meshes of the terrain gave us the visuality that we wanted.
Once we had decided that we were going to use noise, we then started creating the meshes for the terrain. We wanted a section of the environment to feature a very high point that the user could climb so that the project featured a sense of depth in the movement. We created the terrain by creating one mesh for a mountain and two meshes for a lower and smoother terrain that spanned more of the island.
The mesh for the mountain was created by setting the centre vertex's y value as the highest possible value. Then, vertices outside of this centre point were mapped by their x and z coordinates to determine their height. Depending on whether the x and z coordinates were higher or lower than the centre point's x and z coordinate, they were either mapped to a range to gradually increase in height or to a range to gradually decrease in height. We created Perlin noise with 5 octaves to then subtract from the mapped x and z coordinates to create the rough and sloping mountain that we wanted. The smaller parts of the terrain were a lot simpler to create and required using Perlin noise to alter the y values of the vertices and setting starting x and z coordinates to decide where to draw the mesh from.
Other aspects of the build included creating a shader to use as a body of water. In our creative process, we decided that using a shader would be an effect way to perform the vertex manipulation needed to model the movement of water. We experimented with shaders using CodeCircle and then tested them in a provided openFrameworks template. However, when we tried to incorporate the shader into our environment, we realised that the openGL version that we had been using to create parts of the terrain did not correspond to the GLSL version that the shader had been written in. This required converting it from version #150 to #120. Our basic method for creating the shader was to use a variation of noise, using frequency modulation, in the vertex shader to displace the values of y. It also required altering the x and z coordinates with sine and cosine and setting a phase offset based upon these values to alter the way in which the shader appeared to move.
When creating the environment, for ease of use we decided to first create the navigation using keys instead of using the TouchOSC interface with the iPad.
To navigate the camera, we assigned a pointer to the mesh containing all of the terrain vertices. We then created a function that tracked the camera’s current coordinates and checked whether these coordinates matched the coordinates of a specific vertex in the terrain mesh. Initially, we tried to implement this by looping through the mesh vertices and setting the program to search for a vertex with an x and z coordinate that exactly matched the current camera's x and z coordinate. If a match was found, we would then adjust the height of the camera according to the y coordinate of the matched terrain vertex. We realised that this was too exact and did not take account for the “spaces” in between the vertices. From this, we decided to not check for exact matches for x and z coordinates but instead check whether the camera position was in a certain range specified by the distance between the terrain vertices. This problem caused us to realise that we did not consider the navigation enough when building the environment. For example, we had two differently-sized meshes that made up the terrain. Due to the variation in size between vertices, it meant that is was difficult to set an exact range that would be used when checking whether the camera coordinates were within this threshold. Due to the lack of technical thought when putting together the terrain, we had programmed the meshes to overlap each other and did not consider that this would cause a problem. There was a conflict in which vertex was thought to be the correct one when checking against the camera's current position and this led to the wrong height being assigned to the camera's y coordinate. Although these were not huge issues, it did mean that we had to return to alter the design of terrain and adjust the sizes and parameters of them in order for the navigation to be properly implemented. The method used to implement this functionality is shown below.
Once the issues above were solved, we decided that the next logical step was to link the TouchOSC implementation with the navigation with the terrain. However, we then experienced some complications concerning the camera's rotation. When multiple rotations were performed, it would cause the camera's relative axes to change which led to a skewed and unrealistic rotation. We discovered that each rotation affected the camera's Z axis and this would cause the distorted rotation. By retrieving the camera's orientation as an Euler angle and then setting the Z axis to zero after each rotation, it removed the error of the improperly combined rotations.
Once the navigation was complete, we focused our attention to importing the selected models into the environment. Using the ofxAssimpModelLoader add-on, we loaded three-dimensional models into our environment in order to achieve our desired look and feel. Our intention was to either create them with a 3D modelling software or find free and public domain resources online. We wanted to modify the model's positions and vertices through the use of FFT to generate organic movement. The way in which we aimed to achieve this was by manipulating the x, y, or z position of the chosen object with the average of the outgoing signal's frequency. For example, we altered models of teeth to move in the air using a sine wave with a combination of frequency values derived from FFT and the amount of time passed in the program.
We initially had an issue of importing models with their textures and realised that this was because to load models with textures it required .obj and .mtl files. The resources were extremely limited for models that had both the obj and mtl files attached and this prohibited us from implementing a lot of ideas that we had. Along with this, it also meant that we didn’t create as many audio-reactive elements that we had wanted. ‘The Persistence of Sound’ relied upon this audio visual relationship to make an environment that was lively and entertaining.
Evaluation
Overall, we felt that 'The Persistence of Sound' did meet some of the milestones that we had set for it. We felt that some components, such as the checkerboard flooring and the use of an island as the environment, did emanate the aspects of surrealism and isolation that we wanted to. However, we do feel that we did not meet the mark for some aspects of the project. We overestimated the number of resources online for free 3D models and this meant that there were a lot of ideas that we were not able to execute. This in turn meant that we did not emphasise a prominent audio-visual relationship like we had wanted to, and this was something that we had definitely wanted to express. We feel that we did not accomplish this, and that the two do not link in the way that we had hoped.
From a technical perspective, this project has taught us about the components that are needed to create a navigable and realistic environment which at the beginning we were not wholly aware of. For example, we did not realise the amount of planning that was needed in the development of the terrain for the integration of the camera navigation. As well as this, we learnt that simulating lifelike movement is a difficult feat and cannot be achieved solely from the use of the ofCamera and ofNode class functions. For our implementation, it required that we research perspective, Euler angles, and quaternions. Many aspects of the environment, such as setting collision detection for objects and boundaries in which the user can navigate, were features that we had overlooked and underestimated in terms of ease of programming. Overall, we learnt that to create human movement within a program is a delicate and difficult process. The project itself has taught us that careful planning is needed in all stages of development as each will have a direct effect on the other. As we have learnt, superficial planning leads to having to redesign and reevaluate existing work and as a result leads to lost time.
In terms of what we wanted to portray to our audience, we are happy with the overall result of the project. We feel that the use of the iPad with the TouchOSC in conjunction with projecting the game created the immersive experience that we had desired. Most importantly, we feel that we were able to express in some ways the themes present in surrealism through our choice of models, our soundscape, and our use of space and depth within the environment.
References
[1] - Surrealism Art Movement
[2] Sigmund Freud, ‘The Interpretation of Dreams’
[4] Salvador Dali, ‘The Persistence of Memory’
[5] Glenn Brown
[6] Gyuri Lohmuller
[7] Yves Tanguy
[8] Patricio Gonzalez Vivo and Jen Lowe, The Book of Shaders
[9] Another Perspective, ’Butterfly Landscape’
[10] Salvador Dali, ‘Butterfly Landscape (The Great Masturbator In A Surrealist Landscape With D.N.A)