Sketches vs Skeletons

Last month I was in Vancouver at the fantastic MOCO workshop presenting a couple of papers.

The first was called, Sketches vs Skeletons: video annotation can capture what motion capture cannot. It was the outcome of a study we did as part of the Praise project about using technology to give feedback to music learners about their movements and postures. It taught us a lot about the holistic, complex nature of movment, but also about research, being wrong and how to stop being wrong.

We initially made what seemed to be the obvious choice of using motion capture and created a prototype (technology probe) using the kinect, but when we worked with music teachers we discovered that not only was the kinect not sufficient (not particularly surprising), but the whole premise of using skeletal motion capture was misguided.

This really showed us the value of rapid prototyping, we were not particularly attached to our prototype and could recover from it quickly.

Anyway, here is the abstract:

Good posture is vital to successful musical performance and music teachers spend a considerable amount of effort on improving their students’ performance.
This paper presents a user study to evaluate a skeletal motion capture system (based on the Microsoft Kinect for supporting teachers in giving feedback on learner musicians’ posture and movement. The study identified a number of problems with skeletal motion capture that are likely make it unsuitable for this type of feedback: glitches in the capture reduce trust in the system, particularly as the motion data is removed from other contextual cues that could help judge wether it is correct or not; automated feedback can fail to account for the diversity of playing styles required by learners of different physical proportions, and most importantly, the skeleton representation leaves out many cues that are required to detect posture problems in all but the most elementary beginners. The study also included a participatory design stage which resulted in a radically redesigned prototype, which replaced skeletal motion capture with an interface that allows teachers and learners to sketch on video with the support of computer vision tracking.

and this is the full reference:

Sketches vs Skeletons: Video Annotation Can Capture What Motion Capture Cannot

Gillies, Marco , Brenton, Harry , Yee-King, Matthew , Grimalt-Reynes, Andreu and d’Inverno, Mark . 2015. ‘Sketches vs Skeletons: Video Annotation Can Capture What Motion Capture Cannot’. In: Proceedings of the 2nd International Workshop on Movement and Computing. Vancouver, Canada.