In celebration of International Women’s Day on Sunday 8th of March 2020, Nicola Plant, Carlos Gonzalez Diaz and Clarice Hilton from Goldsmiths Computing, ran a special workshop on Using Machine Learning to Design Movement Interaction in VR for a half-day hackathon held at the Tate Exchange. Organised by the people at the XR Diversity Initiative to help make XR more accessible, the event aims to inspire under-represented groups to pursue a career in XR fields and celebrate the stories of women, the female form and movement. There were also workshops teaching 3D sketching in Google Tiltbrush, creating 360 film and non-linear storytelling in immersive technologies. The XRDI hackathon was a part of Digital Maker Collective running a full week of events exploring digital and emerging technologies in the context of arts practice, education, society and the creative industries.
Our participants were a diverse group ranging from theatre directors, visual artists, developers, dance practitioners, academics and more, all interested in learning about how to better bring immersive technologies to their fields.
During the workshop our participants gained a working understanding of how to use machine learning when designing movement interaction for immersive technologies, along with an introduction to VR in Unity. We started with an active, hands-on session exploring full body movement practices to facilitate designing compelling movement techniques. Then dived into learning interactive machine learning concepts and how to apply them to recognise and implement movement interaction designs in immersive environments using our InteractML tool (interactml.com).
To demonstrate the potential of what our participants could achieve with the tool, the team built a special set of VR artworks inspired by female artists that have exhibited their work at the Tate Modern, the setting for our workshop. These included virtual homages to Louise Bourgeois’ ‘Maman’, Kara Walkers ‘Fons Americanus’ and Yoyoi Kusamas ‘Infinity Mirror Room’, each showcasing how various machine learning algorithms can facilitate different movement interaction techniques to control visual elements within each work.
Although many of our participants did not have any previous experience in machine learning or immersive technologies, in a little over three and a half hours our participants were able to implement their movement interaction designs by configuring and training a machine learning model with the tool, allowing them to control light colours, show different animations or control the flow of fountain water just by moving in VR.
This workshop was part of the EPSRC funded research project, 4i: Immersive Interaction design of Indie developers using Interactive machine learning. This is a joint project with University of Coventry, University of the Arts, London, Gibson/Martelli and CodeLiberation.
On Jan 24th – 26th, Goldsmiths Department of Computing, ran a Virtual Reality Hackathon weekend with HTC Vive. Participants were challenged to use VIVE hardware, including Software development kits like eye, lip and hand tracking to create an innovative virtual reality project. Meet some of the Goldsmiths student teams who wowed the judges with their unqiue and brilliant projects.
Group 1: Active Listening Training in VR
(aka the winners of the HTC Vive Hackathon 2020)
“Our team had a very strong technical strength, with extensive knowledge in machine learning, VR and Unity development”
Carlos Gonzalez Diaz
What was the biggest challenge?
For this group the biggest challenge was the use of eye tracking, as well as the additional challenge of adding experimental sensors together. This team experimented with movement, mouth, eye, fingers and EEG (brain electrical activity) trackers. Unfortunately EEG and finger trackers proved too difficult within the strict time restrictions so they dropped them. The team managed to successfully integrate movement, eye and mouth tracking into a machine learning model in the final prototype.
What made the project unique?
The combination of technologies the team used, paired with an interesting story line made the project stand out. The team used the InteractML machine learning tool, an interactive machine learning framework for Unity which was developed by Carlos and colleagues. The machine learning aspect eased the teams workload.
What did they learn?
Cristina Dobre said “I’ve learnt many things from taking part in this event but if I’d choose one, that would be integration-as-you-go. As the team members specialised in different areas and worked in parallel on various parts of the system, we managed to put everything together towards the very end of the event. This gave us only little time to test and fix integration bugs which made the final work very stressful (also given our sleep-deprived states). We managed to have a playable demo with most of the important parts working, but it would have been a much smoother process if the integration would have taken place throughout the development, even though each part might have been only partly finished “
Team members (from left to right in tweet above):
Cristina Dobre, PhD Human Centered AI Characters in VR, Lili Eva Bartha, experienced Designer and Scientist, Claire Wu, PhD Neuroscience, Carlos Gonzalez Diaz, PhD Machine Learning for expressive interactions
Group 2: VR Illusion
This team was a group of Goldsmiths students, many of whom has just started to learn VR in October 2019.
What were the teams strengths?
The skills in the team were varied, Hankun’s knowledge of unity helped them to solve their biggest problem of using C# to set the relationship between the size and position of the object. Yaqi brought skills in 3D modelling, so could quickly create the models they needed. Chaojing is skilled with the storytelling and drawing, so could set the story of the game and draw assets they needed. Finally, Shuai Xu is experienced in user interface design and sourced the music for the project.
How did the project relate studies at Goldsmiths?
Chaojing Li said “For the production of virtual reality games, the sense of the presence of the player is essential, because I think the most important meaning of Virtual Reality is to give people an immersive experience. We think that if there is no such sense of presence, then VR games are no different from games on ordinary platforms. During last semester, in the “3D Virtual Environments and Animation” class, our teacher Xueni Pan and Marco Gillies explained a lot of theory about Virtual Reality and some related psychological knowledge. This gave us a preliminary understanding of how to create a sense of presence in the virtual environment.”
What was the project?
The group focused on virtual reality object interaction and eye tracking technology and how to combine the two to work together. In their programme cubes are thrown onto a mechanical belt like you would see in a factory, the user must stack the cubes onto each other, the challenge is that when the user looks directly at the area where the cubes are their vision is blocked, so they must use their peripheral vision to complete the task.
What is unique about your project
Nima Jamalian said “for our project we reversed the use of eye tracking technology. In majority of application that uses eye tracking the focus is on where user eyes are looking at however in our application we reversed it, the progamme checks if the player is not looking and only then the user can perform the task – so we sort of track where the player is not looking.”
This week we ran our second CareerHack event in partnership with Hacksmiths.
What is CareerHack?
CareerHack is a career & developer event where attendees spend 4 hours competing challenges in teams, showing off their skills to potential employers.
Challenges tested technical skills, with things like building an interactive game, as well as employability skills, like writing a personal profile and skills section for your CV to get students to think about the resources they need when heading into the working world. Employers are there to let the students know more about working at their organisations.
It’s a collaboration between the Department of Computing, Hacksmiths (our student-led tech society), our Careers Department and employers.
We wanted to look at new
ways of employers and students interacting and piloted the event last year to
great success, including one student getting a placement and another applying
successfully for a full time position on graduation.
Employer feedback from last year’s event:
“much more useful than a “traditional” careers fair.”
“being able to watch your students do actual engineering as opposed to just talking about it was really helpful (I was able to flag a number of final year students to our recruitment team as people whose applications should be expedited, if they choose to apply).”
We’re now pulling together feedback from the CareerHack this week and will start planning for next year! Well done to the winning team.
If you’re interested in working with our students on other innovative events, talks, placements and lots more, please contact Eilidh Macdonald.
Push, Pop, Repeat is the first pop-up exhibition of this year’s cohort – presenting works from the MA in Computational Arts and the MA in Independent Games and Playable Experience Design. On Wednesday the 22nd of January 2020, more than 350 visitors have had the opportunity to discover more than 130 artworks.
Most students in the programme had no, or very little, programming experience when they joined in October. After only 10 weeks on the course they are already producing high quality interactive pieces, adding computing to their art practice. The exhibition hosted an incredible variety of works produced at the end of the first term, from generative projection-mapped pieces to physical computing installations.
Selecting works amongst more than 130 pieces is an almost impossible task, but the following pieces particularly grabbed the attention of the visitors. This selection also aims at showcasing the diversity of the projects presented during the pop-up.
Amor Serrano by Alesandra Miro Quesada
Interactive installation – Physical Computing 1
Amor Serrano tells the story of domestic violence in the Andes. Two testimonies of survivors have been recorded and translated from Spanish to English and Quechua (native Andean language). This piece pushes the audience to face the epidemic of feminicide happening in Peru and create discourse on this topic.
Technical details: The circuit includes 6 capacitive sensors, each soldered onto a long copper wire braided inside 6 foot long hair extensions. When touching the hair, it triggers different audio tracks. Each of them narrating a testimony of a domestic abuse survivor.
A cartography of the inside out by Jakob Jennerholm Hammar
Poster – Programming for artists and designers
The program draws structures in time, continually growing over a scarce landscape. A fictional world filled with fictional habitats for fictional inhabitants is created. The work tries out ideas of mapping an interiority and turning it inside out to see what it says about it’s exteriority, or the other way around. Some important influences where gathered from cartography and architectural plans and drawings as a way of understanding and visualizing relationality. Others where graphical notation or visual scoring as a way of working with duration and events in time.
Technical details: The artwork is generated by a program written in Processing. Noise functions and pseudo randomness as well as recursion and fractals are some of the techniques used in the generative process. Printed on A2 sized wallpaper material.
Blurring boundaries. Blurred democracy. Two vibrating motors are connected to the word “democracy” written in copper wire. The motors react to sound. Light is reactive to distance. As you walk away, lights turn chaotic. As you get closer, lights turn on completely. The piece was created in reference to the current Chilean revolution (October 2019-present), which main massage of fight is “Until dignity becomes habit” (“Hasta que la dignidad se haga costumbre”). Until today, serious human rights violations had taken place. Sound recording: “No son 30 pesos, son 46 años”, A las Cacerolas, Colectiva 22bits.
Technical details: The piece is a sound-light interactive sculpture, using 2 vibrating motors, a proximity sensor and led strip lights.
Interactive system and performance – Physical Computing 1 + Programming for artists and designers
_chreeb.dampGrass is a choreographic collaboration between a human user and a virtual dancer. Through the use of a specialist controller and a virtual avatar named “_chreeb”, the creative control of the improvised choreography switches between the human and non-human participants.
Technical details: The system for this performance has been made using a Processing sketch connected to an Arduino controller. The controller itself is made of several arcade buttons, LED’s and toggle switches all housed in a cardboard casing
Tantum Duck is a Hand-controlled Audio-Visual Synthesizer. By mixing computer vision, sound synthesis and mathematical visual shapes the goal was to create an instrument that allows the performer to control both audio and visual parameters at the same time. A constant correlation between the audio and visual outputs is implemented to allow the performer to focus on the sonic and visual details of his/her performance without worrying about keeping them harmonic, as they will always be.
Technical details: The system has been realised in openFrameworks, using computer vision and color tracking to identify the controls and using frequency and amplitude modulation (FM and AM) for the sound synthesis. To follow the chaotic nature of the sounds, the visuals are based on the chaotic map Aizawa Attractor. A chaotic map is a system of differential equations whose apparently-random states of disorder and irregularities are governed by deterministic laws that are highly sensitive to initial conditions, meaning that a small change to any of its coefficients, will result in a substantial change in its output.
Automata by Clémence Debaig
Interactive wearable and performance – Physical Computing 1
AUTOMATA is a performance device allowing the dancer to move elements of the outfit by his/her movements. This piece explores the relationship between puppeteer and object, and the transfer of agency between the two. Is the dancer moving to manipulate the objects or is the movement of the objects influencing the dancer. Who is controlling who?
Technical details: The dancer is wearing an accelerometer on the right arm and another one the left leg. The values received on the accelerometers control the movement of 7 servos placed on the outfit, moving mechanical parts. 3 “flowers” and the left shoulder pad are controlled by the right arm. 2 other “flowers” are controlled by the left leg. Particular attention has been put on the analysis of the acceleration values on each axis (X, Y, Z) to determine with one to use and set different thresholds for each servo.
Drawing machines have been explored for a while but it is still a recurring theme to observe the differences in drawing between Robot and Human. Drawing is a way to express one’s humanity. However, would it be possible for robots to express the essence of being a robot through drawing? This project is presented more as an experiment than an artwork, aiming at reflecting on the meaning of what the essence of a robot is and observing the similarities between human and robot.
Technical details: The drawing machine is using an Arduino and a Processing sketch, connected via serial. The Processing sketch extracts vector points from an SVG file and then sends it to the Arduino to define the position of the pen.
The Chicken of Tomorrow is a projection mapping work devoted to chicken. Combining archive video material and the latest research, it tells the brief story of a bird turned into a commodity. The visuals plays with naïve form and bright colours, yolk yellow, egg white and wattle red. They represent our simple attitude towards chicken. The piece begins with a nostalgic clip of children playing with fluffy chicks. According to studies, we now struggle to consider chicken as animals.
Technical details: The work was created in openFrameworks, mixing generative graphics, typography and videos.
Zala is a Year 1, BSc Creative Computing student at Goldsmiths, University of London.
Progressing from the well-established secondary
school routine to a completely new independent and self-initiated university
one, is a rather big change, but so is moving from a town of 100 000 people to
a city with a population 4-times the size of my whole home country. Here is how
I spend an average weekday in my new life.
Establishing a morning routine
Although my days differ due to my varied timetable, having an established morning routine gives them some structure, which I believe is vital to maintain my productivity and motivation. Since this term none of my classes begin before 12 and I live nearly on campus, I do not have to rush to wake up early in the morning and usually wake up around 8 after, hopefully, a good amount of sleep. I put on a jumper and head straight into our flat kitchen to make some breakfast. After changing and stretching a bit, I opt between going to the library or staying at home to catch up with lectures or work on projects.
Lectures, labs, tutorials
Then come the lectures. After spending the majority of the first term learning the fundamentals of computing and programming, we’re now starting to specialise by taking classes such as Generative Drawing, Graphics, Sounds and Signals and developing an independent creative project, which seems like it’s going to challenge us and our way of thinking quite a bit. As we’re just at the start of the new term, I’m not quite sure what’s in store for us for each specific module, although after having the first lecture of Generative drawing, I’m positive it is going to be one of my favourite classes. For a visual person as I am, Graphics sound interesting as well.
Finishing the Day
As most of my classes finish in the late
afternoon, that does not leave me with much time before the evening. On days
when I do have extra time, I might head to the campus gym, attend a society
meeting/social or go to any event that Goldsmiths’ organises and I find
interesting. By the time I get back, my flatmates are in as well and we spend
our evenings eating and chatting in our kitchen, which serves as our main social
space. We also might head to a pub or on occasion a party. I end my day by
reading, watching a movie or talking with friends and family from back home.
Hey there! I’m Jannat, I’m from Italy, I came to Italy from Bangladesh when I was younger. I am 18 years old and I am currently studying Computer Science at Goldsmiths.
Why I came to Goldsmiths to study Computer Science
Before coming to this fun university, I was an A-level student. I studied Maths, Computer Science, Biology and Italian. It may seem a bit of a weird combination, however, I was interested in finding similarities and differences between Artificial Intelligence and humans so I chose Computer Science and Biology.
I chose to do Maths as I believe it is fundamental when it comes to doing any science or technology subjects. I did Italian because I did not want to forget it if I left Italy and because I love learning new languages.
I decided to study Computer Science for my degree because I have been very interested in tech since I was young, especially programming and AI. I chose to come to Goldsmiths because I felt like they helped me to recognize my potential, even at the interview where we have talked about my A levels, what I’m currently working on and about my future.
What I am studying
My experience so far at Goldsmiths has been amazing. The staff and students are nice and always make me feel included. The course is fun, we get to learn through various activities during our lectures and seminars such as using Kahoot or debugging codes…
This term we have done 4 modules:
Fundamentals of Computer Science
Currently, I am working on my individual website which is worth 60% of my marks. I am making a website about the city of Venice and its history. The content includes advice on the places you should visit and which shops you should check out if you visit. I am also working on the last stage of my games project, which very exciting as the extension requires us to add either an enemy, platform, advanced graphics or sound to our game.
My plan for the future
For the rest of this academic year, we will carry on studying Fundamentals of Computer Science and Programming, and start two new topics, Problem Solving and Symbolic Maths.
Once I graduate, my plan is to work as a programmer, but I would also like to explore different sectors such as cybersecurity and AI. I also aim to visit countries where children receive poor education and do charity work with them as much as I can.
In this post we meet Evan Raskob. Evan runs MA modules in Physical Computing and is pursuing a PhD in “liveness in physical computational art & design”. He also performs as a “livecoding” musician under the name BITLIP.
Evan originally hails from the picturesque town of Mahopac in New York State. His father was a dentist and his first interest in computing sparked from reading computing magazines in his office waiting room. At high school, he was inspired by a friend’s father who worked at IBM to use his burgeoning technology skills to create text based adventure games for other students, in the manner of the text-only BBS games that were popular at the time like Legend of the Rd Dragon (LoTRD) and others. He then managed to land himself a summer internship at Bell Labs in New Jersey, the iconic home of the creators of C++ and birthplace of both transistors and the laser.
Evan went to study at Cornell, then moved away from small town life and relocated to New York City for postgraduate study on the Interactive Telecommunication Program (ITP) at NYU. During this time, he was also involved with Julie Martin and Robert Whitman, some of the remaining founding members of Experiments in Art and Technology (E.A.T), an influential and forward-thinking collective of artists and engineers. He immersed himself in creative interactive technology and became particularly interested in building his own digital instruments and audio-visual “controllers”.
His passion in making his own musical, digital DIY/punk instruments has continued. He performs under the artist name BITLIP, where he explores making music through live coding and particularly using 3D printers as instruments. In 2019 Evan was quoted in the New York Times about his live coding:
“The great thing about punk is they played three chords, and you said, ‘I could do that,’” Evan said. “With live coding, you type in a few lines and hit compile and you’re playing music.”
This experience has shaped his work in the UK. Evan relocated to London in 2006 and has since worked at the University of Creative Arts, Kingston University and Ravensbourne University teaching and developing courses in Design, Coding and Computer Games. He then went on to work at the Royal College of Art, where he ran the “Mixed Reality Design” theme exploring VR/AR and sensory design.
Evan joined Computing at Goldsmiths in 2018. He lectures in Physical Computing and takes part in teaching trips to China. At the same time he is working toward a PhD titled “liveness in physical computational art & design”. In particular, he is looking at live computational and procedural sculpting with computer numerical control and 3D printing, augmented by virtual reality/augmented reality and machine learning.