being

by: Kotryna Sajevičiūtė

Introduction

     This year, I have decided to create a real time projection mapping. A year ago, I came across Kat Von D’s performance with the Wildbytes (a real time projection mapping on the face performance) and wanted to improve my knowledge in programming by trying it out as well. After my trip to Venice, Italy, where masks are worn by mostly every traveler, I came up with the idea to generate masks that would be projected on the face and bring the positive and festive mood to the exhibition. One of the reasons I chose to do that is because while studying and working in London everyone seems to be mainly surrounded by the crowds, with the negative aura and tiredness and always being in a hurry. I have also realised that a handful of artists make a depressive and negative art. After stress and little anxiety I started to come up with negative ideas for the art piece as well, but to fight through that I have decided to make something bright and bring a little joy to the small corner in the gallery. The piece was interactive and anyone in the gallery was able to try those digital masks out.
     The name of the art piece is ‘being’. The term means the existence, the nature or essence of a person or a real or imaginary living creature or entity. I chose this word as a name to represent the current moment - existence. People wear masks on Halloween and other celebrations as a disguise to represent some kind of creature. This word has a lot of meanings, that represents my piece.

Live Projection Mapping with Kat Von D and Wildbytes

     I have been doing photography for most of my life and always loved taking pictures of random people, especially doing portraits. I believe that face and eyes are the most intimate parts of the human’s body. People tend to hide themselves and not always express their thoughts or feel self-conscious when being in the centre of attention. By creating digital masks and letting anyone in ‘put them on’ I was aiming to make people become a part of the art piece and to be on the centre of the attention. Even though some might feel uncomfortable, at the same time, having a digital mask on their face would surprise the one and make one feel excited. When tourists come to Venice they wear the brightest and colorful masks to walk around the city even though they would not do that in everyday life. Likewise the digital masks -- it’s just a short “holiday” spirit.

Window Shocase of masks in Venice and artist making masks

The concept

     The masks that were projected on the face were the symbols of celebrating life, self realisation, universe and the birth. I have drawn sugar skulls, which are used in Mexico to celebrate “The Day of Dead”. Bright colors in the skulls reflects life, and the celebration celebrates just that. I have generated some mandalas and tree with OpenFrameworks. Mandala is a spiritual and ritual symbol in Hinduism and Buddhism, which represents the universe -- Seeing them grow and rotate is very soothing. Another symbol I have generated was a growing tree, the tree of life, which is a spiritual symbol, that is inspirational. I have also tried to center the mandalas to the center of the forehead as it represents Bindu. Bindu -- is the point or dot around which the mandala is created, representing the universe. . Bindu is considered the point at which creation begins and may become unity. Whenever you have it on you are automatically get counsious of your surroundings. The reason I chose these symbols are because they represent the beauty of life, appreciation of being alive and brings the positivity to the inner self. That is what was represented within my art piece.

Sugar Skulls by Sarah Forsterling 

Some sugar skulls I have drawn to use for a projection

Technical Research

     For the best accuracy to track people I have decided to make a project using a Kinect Xbox. For the face tracking I have used Kyle’s McDonald’s addon ofxFaceTracker which tracks the face features using web camera. The projection mapping was made with the projector according to the tracked object where I have used the addon ofxKinectProjectorToolkit. This addon is used to calibrate the projector with the Kinect, it generates an xml file and works perfectly to make Kinect understand the same coordinates as a projector has. At first I have explored the Face Tracker addon and made some prototypes with it.

Testing the Face Tracker Addon

     Furthermore, I have experimented with the Kinect Projector Toolkit addon too see if it accurate enough to track and project something on the person. After few tries, and trying to calibrate it I made it work.

     I then decided that it would be a great idea to combine both addons and see if it works well.
     This is a snippet of code that was used with the ofxFaceTracker. It creates a polyline and tracks the selected face feature (right eye in this case). Then I had to translate it to the certain position so that whenever you move your head the drawing would follow the right eye.

     This is a snippet from the ofxKinectProjectorToolkit. It gets contour, label, center point, and age of contour. It places a contour into a vector called points. In the for loop it gets the tracked 3D coordinates of a Kinect and projector coordinates that are read from the xml file of calibration done before (variable called kpt). It multiplies by the projector resolution and projected point and projects the pattern onto the tracked body.


     
     This is how I combined both addons. I have declared same polyline that is going to track a left eye. Instead of getting world coordinates from a vector I replaced them with the polyline Face Tracker x and y coordinates. There is a better way to do it though, because by changing this code it makes unnecessary use a Kinect.

     But the code worked fine and I managed to make a projection align on the face.

Process and the problems I have encountered

     I have started experimenting with various implementations what could be displayed and projected on one’s face. I have found a couple of great examples made of meshes that applied on the face and mapped perfectly on the face. These examples were used only as an idea generator, and no code was implemented within my project.  I just wanted to create my own and unique project. After a while I have encountered another issue, the projector I was working with stopped working so I had to use another one. The problem with that projector was that it had a very shallow depth of field, which meant that I could not let a person move around much because the projection would be unfocused at certain points. In addidion, I didn't think through that using a tracker with the web camera need a better lighting. Instead of using face tracker addon I could have tracked certain point with the Kinect and made the visuals appear on the certain position from that point.
     One more problem I have faced was the issue with the ofxGif addon. I have drawn some skulls with Adobe Photoshop and to display them I have used the ofxGif addon, which made the animated images very glitchy and moved them away from the set position. These problems would have been easily fixed if I would have noticed this issue before. The book I am reading now taught me that “...even the incidental effects of the processes of Nature have their own charm and attraction.” This boosted my confidence and helped me see my ‘broken’ and not perfect piece in a different point of view and made me happy with the result as it is.
     Furthermore, doing a real time projection mapping with the Kinect is a bit risky. Nonetheless, I have taped the Kinect to a projector, it was not tight enough. After multiple people have walked around my piece, the vibration caused by their movement dislocated the Kinect and moved away from the projector -- becoming uncentered. Causing the visuals to become less and less accurate.
     My experience during the exhibition has demonstrated some improvements. I have noticed that some people were confused and unable to figure out how my art piece works without my assistance/without my presence. Therefore, looking back, having placed a mirror for the viewers would have provided a more understandable and pleasant experience for the viewers.

Preparing for the exhibition (photo taken from Twitter @GoldsmithsUoL)

Conclusion

     All in all, I had some issues that I wasn’t able to fix until exhibition I found that some people have expected a more professional piece. Nevertheless, I found the audience enjoying my piece and their experience while having or seeing visuals on their face. There are things that I would have done differently to make my art piece look more professional and accurate in terms of projection mapping. Working on this project was proved to be great experience as I improved my knowledge in c++, Kinect Xbox, projection mapping and coding in general. I don't think I would work with the Kinect in the future as it was a bit too complicated and no matter how hard I tried I still did not get the outcome I was expecting to get. I am glad that I have managed to make a projection mapping and I believe that in the future I could work out better projects.

Some images from the exhibition

Gitlab Repository
           http://ksaje001@gitlab.doc.gold.ac.uk/ksaje001/ProjectionMapping.git

References

• ​​Sugar skulls and the Day of Dead

               https://sarahforsterling.com/2015/02/07/the-sense-of-life-and-death-in-mexico/
               https://www.popsugar.com/latina/What-Meaning-Behind-Day-Dead-Sugar-Skulls-42083035

• ​Wildbytes and Kat Von D

                http://wildbytes.cc/
                https://vimeo.com/143267919

• Bindu and Mandalas 

                http://www.chinabuddhismencyclopedia.com/en/index.php/Bindu_(symbol)
                http://www.whats-your-sign.com/meaning-of-mandala.html