Date of Original Version




Rights Management

All Rights Reserved

Abstract or Description

Computing systems have traditionally relied on purely quantitative means of assessing users. Software may record facts about users such as their click behaviors, their accuracies, and their rates of recurrence, and may then use this acquired data to make predictions about their behaviors and preferences. However, users’ inputs into a machine, sequences of key presses and mouse movements, do not capture all of their desires and needs. The machine may never know that a particular user enjoys the glow of a button and its placement among others elements of an interface. Furthermore, it cannot perceive a user’s discomfort during accomplishing a task; it may simply translate the user’s high accuracy in hitting various steps of a process into a mastery of the task. The machine, which might be intelligent, may not sense the user’s uncertainty of his or her actions, expressed by widened eyes, a stutter in voice, and a hand clasped behind the neck.

In this project, we explore how an affect-mediated system, a computing system that adapts its actions and behavior to the emotional state of its users, can improve their abilities to complete tasks and meet their goals. In particular, we apply facial expression recognition, one method for estimating a human’s emotional state, to a children’s game, giving it the ability to adjust its difficulty based on its player’s perceived unease. Through experimentation with this game, we determine whether affect-mediation helps users achieve their goal of winning the game, and whether in general affect-mediated systems can aid in user task completion.

Our study consisted of two conditions: affect (the game would adjust its difficulty based on the player’s emotion) and control. Participants in the affect condition tended to have a higher catch/total ratio (the number of items caught by the participant divided by the total number of items spawned in the game). They also had less struggles (a catch followed by one or several misses) and these struggles were more spread out than in the control condition, suggesting a higher level of engagement.


Advisor: Raja Sooriamurthi

Department of Information Systems