Fluid Gesture Interaction Design

The GIDE gesture design interface

Fluid gesture interaction design: applications of continuous recognition for the design of modern gestural interfaces, Bruno Zamborlin’s new paper (that I helped on) is about to be published, you can access it on the Goldsmiths Repository:


The paper is based on Frédéric Bevilacqua’s Gesture Following algorithm for continuous gestures recognition (which Bruno worked on), but in this worked we really looked carefully at the HCI of gesture interface design. If you want people to design good gesture interfaces it isn’t enough to have good gesture recognition software, you need design tools that support them in doing so. In particular you need to support them in tweaking parameters to get optimal performance and help them know what to do when things don’t work as expected. Bruno showed in this paper, how real time visual and auditory feedback about the recognition process can help people design better interfaces more quickly.

facebooktwittergoogle_plusredditpinterestlinkedinmailby feather
Posted in Uncategorized | Leave a comment

Kinect can open up a new world of games customization

The Microsoft Kinect is the device that has promised to change the way we play games and interact with computers by making real time motion tracking possible on commodity hardware, but it’s potential doesn’t stop there. We’ve been exploring how it can massively expand the way players can customise their games.

Customisation is a big part of modern gaming, particularly in Massively Multiplayer Online games, where players customise their avatars to develop an individual identity within the game, and communicate that identity to other players. Up to now customisation has mostly been about changing how characters look, but that is only one aspect of what makes a character unique. How a character moves is also very important. Even more fundamentally we could customise how characters respond to events in the game, what game developers call Artificial Intelligence. Up to now customising these would involve complex animation and programming, skills that ordinary players don’t have. With Andrea Kleinsmith, I’ve been exploring how motion tracking like the kinect can make customising animtaion and AI easy. Players can use their own movements to make the animations for the characters. AI is harder, but we’ve been looking at how machine learning to build AI customisation tools. Rather than have to program the AI, players can act out examples of behavior using motion capture or a kinect, and our machine learning algorithms can infer AI rules to control the character.

We’ve recently had a paper published in the International Journal of Human-Computer Studies that describes a study we did that allowed players to customise thier avatars’ behaviour when they win or loose a point in a 3D version of the classic video game Pong. You can see it here:

Kleinsmith, Andrea and Gillies, Marco. 2013. Customizing by Doing for Responsive Video Game Characters. international journal of human-computer studies, 71(7), pp. 775-784. ISSN 1071-5819

facebooktwittergoogle_plusredditpinterestlinkedinmailby feather
Posted in Uncategorized | Leave a comment

Creative Programming for Digital Media & Mobile Apps

Or MOOCpocalypse Now.


After months of hard work and tight deadlines our Massively Open Online Course, Creative Programming for Digital Media & Mobile Apps, will be launching on Coursera on Monday with over 70 000 students already enrolled. The course will:

teach you how to develop and apply programming skills to creative work. This is an important skill within the development of creative mobile applications, digital music and video games. It will teach technical skills needed to write software that make use of images, audio and graphics, and will concentrate on the application of these skills to creative projects.  Additional resources will be provided for students with no programming background.

The course is being taught be Mick Grierson, Matthew Yee-King and me.

Thanks to everyone who has supported us, including Niklaas van Poortvliet of UCL Publications and Marketing Services (PAMS) for the epic amount of work he and his team have put in to produce the fantastic videos. Barney Grainger and Michael Kerrison of the University of London International Academy for all the help and support they have given through the creation of this course. And of course all the current and former Goldsmiths students who will be helping support you on the forums: Vlad Voina, Tom Rushmore, Will Gallia and Joe Boston.




facebooktwittergoogle_plusredditpinterestlinkedinmailby feather
Posted in Uncategorized | Leave a comment

Goldsmiths Masterclass

Welcome to everyone attending our masters classes at Goldsmiths this week.

You can see our schedule here:


Today and tomorrow I’ll be running the computing masterclasses and will be looking at some of the exciting new web technologies that have been developed in recent years. The main focus will be on HTML5. We will be using some of the HTML5 example developed at Goldsmiths that you can access here:


We will also introduce Processing, a great programming environment for rich media interactive web sites, and the main teaching language we use in first year at Goldsmiths. You can download Processing here:


and the documentation is here:


If you are interested you can look at some examples.  I will be showing this today:



and this is a great resource of processing examples:



facebooktwittergoogle_plusredditpinterestlinkedinmailby feather
Posted in Uncategorized | Leave a comment

Bruno Latour’s challenges for CHI

Bruno Latour just gave the closing keynote for CHI 2013 and he issued four challenges for HCI research. I thought I would get down some thoughts about them before I forget.

Before I talk about the challenges, I should try to describe his central theme. He was arguing against the division of sociology into two scales the unconnected individual and the unindividualised collective. He instead argues that we should think in terms of overlapping and interconnecting “monads” (I won’t try to explain the term). He thinks that digital technology can help to analyse data without going to the two poles of individual qualitative datum or collective, aggregate statistics.

This kind of aligns with my thoughts that interactive machine learning could help to bridge this divide by having human interaction that focuses on the detail of different aspects and items of data within the statistical analysis of machine learning (this is still very vague on my part, but I think there is something there and maybe Bruno Latour does too).

Overall, Latour wants the CHI community to help break down the individual/collective polarity, but in particular he issued four challenges:

Getting rid of data

His first challenge was to help get rid of data from large data sets (presumably so you are only left with “interesting” data in some sense). Given the rest of this talk I interpret this not as wanting to focus on individual items but to pick up connected elements that are important without either aggregating all of the data or removing them from their connections with the rest of the dataset. I can imagine that there could be powerful tool that allows researchers to investigating small snippets of data while a statistical engine runs in the background, clustering or otherwise picking out connections between those snippets and the rest of the dataset.

Capturing the inner narrativity of overlapping monads

I will have to think about this one, but it is something about bridging the gap between human narrative and statistical analysis. He referred to data journalism and how data is used in both interactive and narrative contexts in things like the guardian coverage of the london riots.

Visualizing heritage, process and genealogy

How to visualise these temporal qualities without relying in static structure or loosing connectedness. While answering questions he stressed the importance of not falling back on unchanging structures but acknowledging the changing nature of monads and their connections. This seems to me to relate to another theme that came up quite a lot in CHI (from Bill Buxton to the NIME SIG), the need to have time as a first class concept. This would make it possible to model the evolution of data without relying on static structures (maybe).

Replacing model building and emergent structure by highlighting differently overlapping monads. 

I guess that this would require a very dynamic analysis that made it possible to apply many different and changing models to data. I think that interactive machine learning could help a lot here by using the human element to navigate different interpretations, learnt models and views of the data.


Nate Matias has a much more accurate write up of the challenges. I’ll quote them below, but I’ll just warn that I think he’s not quite right about collective phenomena. He says that “Collective phenomena grow out of these collecting sites”, but this isn’t quite what Latour is saying, after all collective phenomena don’t exist so they can’t emerge. Latour is from a Science Studies background so when he talks about collecting sites he is thinking about scientific data collection instruments (or their aggregations) like microscopes, telescopes, mass spectrometers, semi-structured interviews or surveys. While we might think of a survey as collective and an interview as individual they are both just methods of collecting data which both observe and transform (“perform”) the data thus creating a particular view on the world. There is no innate distinction between individual and collective phenomena in the world, just phenomena that are created by methods of data collection (or more precisely by their interaction with the world). This means that there isn’t a division into two (or more scales) collective and individual but a mass of different views of the world, each specific to it’s data collection method.

Anyway, that small criticism aside, well done to Nate, for the otherwise excellent explanation, here is his summary of the challenges:

Visual complexity produces opacity. Massive individualizing data produces beautiful, playful hairballs which show us nothing. How do we get filter and focus data while still appreciating monads?

How can we capture the inner narrativity of overlapping monads? Latour shows us the “512 paths to the White House” visualization by Mike Bostock and Shan Carter. The other, the Guardian’s Rumour tracker, following the 2011 London riots. The idea that quantitative is different from qualitative is an artifact of the history of social science and a fallacy arising from the distinction between the individual and the collective, he tells us.

How can we visualize heritage, process, and genealogies? Latour shows us a paper he worked with on “complex systems science” (I couldn’t find it). To be a monad is to establish connections, but timeseries visualizations can focus on structure rather than connectedness (like the paper on Phylomemetic Patterns in Science Evolution by Chavalarias, Cointet et al)

How can we replace models about emergent structures with models that highlight differentially overlapping monads? He shows us a hairball network diagram and talks about the difficulty of moving beyond the hairball to understand the overlapping monads



facebooktwittergoogle_plusredditpinterestlinkedinmailby feather
Posted in Uncategorized | Leave a comment


The Embodied Audio-Visual Interaction group and Goldsmiths’ Computing generally is going to be out in full force this year at CHI 2013 in Paris. See you all there (I’m arriving monday).

Here is a list of our presentations:

Caramiaux, Tanaka  – Beyond Recognition (Alt.CHI)  Wed 01/5 9:00am


Hazelden, Yee King, D’Inverno – WeCurate (Work in Progress) Wed 01/5?


Kiefer, Grierson  - Squeezable interface (Interactivity Explorations)


Seipp, Devlin – One-handed Website (Interactivity Research)


Pachet, Roy, d’Inverno – Reflexive Loopers (Note) Wed 01/5 9:00am


Bevilacqua, Tanaka, et al – SIG NIME (SIG)  Wed 01/5 2pm


facebooktwittergoogle_plusredditpinterestlinkedinmailby feather
Posted in Uncategorized | Leave a comment

Mogees at TEDx Brussels

Bruno Zamborlin shows off Mogees at TEDx Brussels.

(Also featuring Steph Horak from Goldsmiths)

facebooktwittergoogle_plusredditpinterestlinkedinmailby feather
Posted in Uncategorized | Leave a comment

Actors teach game characters the subtleties of body language

A nice wired article by Liat Clark, about our project:



facebooktwittergoogle_plusredditpinterestlinkedinmailby feather
Posted in Uncategorized | Leave a comment

Actors help university create body recognition technology

An article about my research in Develop:


facebooktwittergoogle_plusredditpinterestlinkedinmailby feather
Posted in Uncategorized | Leave a comment

Computação criativa


Creative Computing in Brazil.

My colleague Edison Puig Maldonado has been running workshops in Creative Computing at FAAP in Sao Paolo:


facebooktwittergoogle_plusredditpinterestlinkedinmailby feather
Posted in Uncategorized | Leave a comment