{"id":2792,"date":"2019-10-17T07:53:36","date_gmt":"2019-10-17T07:53:36","guid":{"rendered":"http:\/\/www.doc.gold.ac.uk\/blog\/?p=2792"},"modified":"2019-10-17T07:53:36","modified_gmt":"2019-10-17T07:53:36","slug":"assessment-for-gamers","status":"publish","type":"post","link":"http:\/\/www.doc.gold.ac.uk\/blog\/?p=2792","title":{"rendered":"Assessment for gamers"},"content":{"rendered":"\n<p><b>Dr Simon Katan, Director of Undergraduate Studies, recently published this Medium post on the gamification of learning. We reprint it here.<\/b><\/p>\n\n\n\n<p>When I was 14 I took my French GCSE. My French teacher, Madame Percival, had studied the exam and mark scheme intricately. She taught us just the right grammatical constructions to achieve the highest marks.<em> \u2018Au bord de la mer\u2019<\/em> was worth a particularly high number of marks, so we were all to use this phrase as much as possible.<\/p>\n\n\n\n<p>In my exam I stitched together these little memorised passages to score maximum points \u2014 <em>\u2018Hier soir, je suis all\u00e9 \u00e0 la cr\u00eaperie et j\u2019ai choisi une cr\u00eape au jambon mais c\u2019\u00e9tait trop sal\u00e9, j\u2019ai d\u00e9cid\u00e9 d\u2019acheter un verre d\u2019eau<\/em> \u2026 oh yeah \u2026<em> au bord de la mer.\u2019<\/em> <\/p>\n\n\n\n<p>I got an A. I can\u2019t speak French. I\u2019m not proud.<\/p>\n\n\n\n<p>Such behaviour is called \u2018gaming the system\u2019 and it is the scourge of university lecturers everywhere. In Introductionary Programming it takes numerous forms ranging from the pragmatic to the malevolent. Some examples are selectively attempting assignments to achieve a minimum pass, reverse engineering projects around grading criteria, abstaining from programming roles in group work, exam cramming, manipulating teaching assistants to write their code, superficially adapting copied code, and sharing exercise solutions on Whats App groups.<\/p>\n\n\n\n<p>The problem of course with these behaviours is that they result in poorer learning. Whilst we can adapt our assignments to close loopholes and can reprimand and inform those we catch in the act of gaming, a significant portion of students persist. Achieving a minimum pass at introductory programming with scant knowledge of how to program a computer, such students find themselves facing increasingly insurmountable challenges as their course attempts to scaffold on faulty foundational knowledge. <\/p>\n\n\n\n<p>I\u2019m in no doubt that this scenario is a major contributor to Computer Science\u2019s status as the worst performing subject with regards to undergraduate non-continuation in the UK.<\/p>\n\n\n\n<p>This isn\u2019t the fault of our students. It\u2019s not surprising that they game the system in this way. Our students are gamers. Outside of education, they have been brought up on a rich diet of commercial video games informed by forty years of industry experience in optimising for maximal engagement. <\/p>\n\n\n\n<p>This cultural fact is immutable, but we needn\u2019t view it negatively. Jane McGonigal says that \u201c\u2026 when we\u2019re in game worlds, I believe that many of us become the best version of ourselves \u2014 the most likely to help at a moment\u2019s notice, the most likely to stick with a problem as long at it takes, to get up after failure and try again.\u201d<\/p>\n\n\n\n<iframe loading=\"lazy\" width=\"560\" height=\"315\" src=\"https:\/\/www.youtube.com\/embed\/dE1DuBesGYM\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen><\/iframe>\n\n\n\n<p> Viewed through the eyes of our students, university degrees look just like games; they have rules, challenges and goals, points, levels and competition. The problem is that they\u2019re bad games. Just imagine if computer games functioned like exams. Super Mario Bros with no feedback, no stats, no lives, and no replays probably wouldn\u2019t be so much fun.<\/p>\n\n\n\n<p>It is with all this in mind that in 2016 I first began work on developing gamified assessment for teaching programming rudiments. Over three years, in collaboration with my colleague <a href=\"https:\/\/www.gold.ac.uk\/computing\/people\/anstead-edward\/\">Edward Anstead<\/a>, we have developed Sleuth, a series of film-noir gamified code puzzles. Students access Sleuth via a web-app themed as a detective agency \u2018Sleuth &amp; Co\u2019 in with them playing the character of a fledgling detective. They are guided by \u2018the Chief\u2019 who gives them feedback on individual puzzle attempts as well as their general progress in the game. <a href=\"http:\/\/doc.gold.ac.uk\/www\/340\">You can give it a go here<\/a> .<\/p>\n\n\n\n<p>Our design uses procedural content generation to provide students with as many opportunities to practice their coding rudiments as possible, and uses simple game dynamics to encourage them to do so. A key feature is instantaneous feedback. Students can upload a puzzle attempt at any time for immediate grading and feedback from the Chief. Students get five goes at solving a particular puzzle before the Chief suspends them from the case and prescribes some cool down time. <\/p>\n\n\n\n<p>However, such suspensions have no penalties attached \u2014 we want our students to try and fail as many times as they need to achieve mastery. On their return from the cool down period , students are presented with a procedurally generated variation of the puzzle which they can attempt afresh. As students progressively solve puzzles, so they see their score increase in real time. In negating any opportunity for over-estimation of their performance such mechanics put students firmly in control of their final grade.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" width=\"960\" height=\"284\" src=\"https:\/\/www.doc.gold.ac.uk\/blog\/wp-content\/uploads\/2019\/10\/sleuth.png\" alt=\"\" class=\"wp-image-2793\" srcset=\"http:\/\/www.doc.gold.ac.uk\/blog\/wp-content\/uploads\/2019\/10\/sleuth.png 960w, http:\/\/www.doc.gold.ac.uk\/blog\/wp-content\/uploads\/2019\/10\/sleuth-300x89.png 300w, http:\/\/www.doc.gold.ac.uk\/blog\/wp-content\/uploads\/2019\/10\/sleuth-768x227.png 768w\" sizes=\"(max-width: 960px) 100vw, 960px\" \/><figcaption><em>Sleuth<\/em> case 302: Bank heist<\/figcaption><\/figure>\n\n\n\n<p>The environment we have created has undoubtedly fulfilled its primary aims of motivating and facilitating practice. In the initial on-campus run students made a total of 42534 code submissions \u2014 an average of 138 per student over a ten week period. Despite perceiving the task\u2019s level as between fair and difficult, the class\u2019 achievement was high with an average grade of 90.67%. We\u2019ve now had over 2500 students play Sleuth both on campus and <a href=\"https:\/\/www.coursera.org\/lecture\/introduction-to-computer-programming\/getting-started-with-sleuth-rMaGL\">online with Coursera<\/a>, and we\u2019ve had similar responses across the different scenarios.<\/p>\n\n\n\n<p>However, Sleuth has also engendered some unexpected and somewhat dysfunctional behaviours. Sleuth has a partially open level design which allows students to progress in a non-linear fashion. In designing this we imagined students setting aside levels which they found difficult, and returning once they had built confidence on other levels.<\/p>\n\n\n\n<p>Contrary to our expectations, students make little strategic use of this design. The majority progress doggedly in sequence often at the cost of many failed attempts at the harder levels as they fixate on individual problems. The resultant frustration finds its expression through increasingly angry VLE forum posts as deadlines loom.<\/p>\n\n\n\n<p>This obsessive behaviour also carries over into attitudes about grades. Much to our surprise, despite a pass threshold of 40% and a first class threshold of 70%, around a third of students expect, indeed demand, a grade of 100%. I have found myself dealing with student demands for deadline extensions to increase their grade from 85% to 100% and late night angry emails from students with a grade of 97% who can\u2019t solve the final puzzle. <\/p>\n\n\n\n<p>This is borne out in the final grade distribution which, as opposed to being normally distributed or bimodal, peaks sharply at 40%, 70%, and 100%. We could characterise the students at these peaks as being respectively pass-orientated, grade-orientated, and game-oriented.<\/p>\n\n\n\n<p>All of this raises quite a few dilemmas for Edward and me in where to go next with Sleuth. Our encounters with obsessive behavioural patterns might tempt us to iterate on our game design to engineer them away, but in doing so do we risk robbing students of an opportunity to develop autonomy ? How as pedagogues can we be sure that the behaviours we are engineering are more desirable than others ?<\/p>\n\n\n\n<p>Similarly the strangeness of our grade distributions could lead us to raise the difficulty of some puzzles or the use of other available metrics to improve grade differentiation, but what pedagogical or behavioural improvement would such a change serve ? <\/p>\n\n\n\n<p>For me, such questions expose the contradictions at the heart of current approaches to assessment in higher education. Game mechanics are a powerful tool for motivating shifts in student behaviour, but be warned, such power has disruptive potential.<\/p>\n\n\n\n<ul><li><a href=\"https:\/\/medium.com\/@kimonsatan\/assessment-for-gamers-7edafac3cdea\">Originally published on 13 October 2019 on Medium<\/a><\/li><li><a href=\"https:\/\/en.wikipedia.org\/wiki\/Gamification_of_learning\">Wikipedia: Gamification of learning<\/a><\/li><li><a href=\"https:\/\/www.gold.ac.uk\/computing\/studywithus\/\">Study Computing at Goldsmiths<\/a><\/li><\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Dr Simon Katan, Director of Undergraduate Studies, recently published this Medium post on the gamification of learning. We reprint it here. When I was 14 I took my French GCSE. My French teacher, Madame Percival, had studied the exam and mark scheme intricately. She taught us just the right grammatical constructions to achieve the highest &hellip; <a href=\"http:\/\/www.doc.gold.ac.uk\/blog\/?p=2792\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">Assessment for gamers<\/span> <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":9,"featured_media":2794,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"spay_email":""},"categories":[107],"tags":[],"jetpack_featured_media_url":"http:\/\/www.doc.gold.ac.uk\/blog\/wp-content\/uploads\/2019\/10\/gamerz.png","_links":{"self":[{"href":"http:\/\/www.doc.gold.ac.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/2792"}],"collection":[{"href":"http:\/\/www.doc.gold.ac.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.doc.gold.ac.uk\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.doc.gold.ac.uk\/blog\/index.php?rest_route=\/wp\/v2\/users\/9"}],"replies":[{"embeddable":true,"href":"http:\/\/www.doc.gold.ac.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2792"}],"version-history":[{"count":1,"href":"http:\/\/www.doc.gold.ac.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/2792\/revisions"}],"predecessor-version":[{"id":2795,"href":"http:\/\/www.doc.gold.ac.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/2792\/revisions\/2795"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/www.doc.gold.ac.uk\/blog\/index.php?rest_route=\/wp\/v2\/media\/2794"}],"wp:attachment":[{"href":"http:\/\/www.doc.gold.ac.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2792"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.doc.gold.ac.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2792"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.doc.gold.ac.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2792"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}