Analyses of user experience in the electronic entertainment industry currently rely on self-reporting methods, such as surveys, ratings, focus group interviews, etc. We argue that self-reporting alone carries inherent problems-mainly the misinterpretation and temporal delay during longer experiments-and therefore, should not be used as a sole metric. To tackle this problem, we propose the possibility of modeling consumer experience using psychophysiological measures and demonstrate how such models can be trained using machine learning methods. We use a machine learning approach to model user experience using real-time data produced by the autonomic nervous system and involuntary psychophysiological responses. Multiple psychophysiological measures, such as heart rate, electrodermal activity, and respiratory activity, have been used in combination with self-reporting to prepare training sets for machine learning algorithms. The training data was collected from 31 participants during hour-long experiment sessions, where they played multiple video-games. Afterwards, we trained and compared the results of four different machine learning models, out of which the best one produced ∼96% accuracy. The results suggest that psychophysiological measures can indeed be used to assess the enjoyment of digital entertainment consumers.

Department of Computer Science Czech Technical University Prague 166 36 Prague Czech Republic

Department of Cybernetics and Artificial Intelligence Technical University in Košice Letná 9 040 01 Košice Slovakia

The Biorobotics Institute Scuola Superiore Sant'Anna 560 25 Pisa Italy

Zobrazit více v PubMed

Sales, Demographics and Usage Data: Essential Facts about the Computer and Video-Game Industry. [(accessed on 25 February 2019)]; Available online: http://www.theesa.com/wp-content/uploads/2017/09/EF2017_Design_FinalDigital.pdf.

Bernhaupt R. Game User Experience Evaluation. Springer; Berlin/Heidelberg, Germany: 2015.

Desmet P.M., Hekkert P. Framework of product experience. Int. J. Des. 2007;1:57–66.

Hassenzahl M., Tractinsky N. User experience—A research agenda. Behav. Inf. Technol. 2006;25:91–97. doi: 10.1080/01449290500330331. DOI

Law E.L.C., Roto V., Hassenzahl M., Vermeeren A.P., Kort J. Understanding, scoping and defining user experience: A survey approach; Proceedings of the SIGCHI Conference on Human Factors in Computing Systems; Boston, MA, USA. 4–9 April 2009; pp. 719–728.

Lallemand C., Gronier G., Koenig V. User experience: A concept without consensus? Exploring practitioners’ perspectives through an international survey. Comput. Hum. Behav. 2015;43:35–48. doi: 10.1016/j.chb.2014.10.048. DOI

Hassenzahl M. User experience (UX): Towards an experiential perspective on product quality; Proceedings of the 20th Conference on l’Interaction Homme-Machine; Metz, France. 2–5 September 2008; pp. 11–15.

Vorderer P., Bryant J. Playing Video Games: Motives, Responses, and Consequences. Routledge; New York, NY, USA: 2012.

Ryan R.M., Rigby C.S., Przybylski A. The motivational pull of video games: A self-determination theory approach. Motiv. Emot. 2006;30:344–360. doi: 10.1007/s11031-006-9051-8. DOI

Tamborini R., Bowman N.D., Eden A., Grizzard M., Organ A. Defining media enjoyment as the satisfaction of intrinsic needs. J. Commun. 2010;60:758–777. doi: 10.1111/j.1460-2466.2010.01513.x. DOI

Tamborini R., Grizzard M., David Bowman N., Reinecke L., Lewis R.J., Eden A. Media enjoyment as need satisfaction: The contribution of hedonic and nonhedonic needs. J. Commun. 2011;61:1025–1042. doi: 10.1111/j.1460-2466.2011.01593.x. DOI

Kivikangas J.M., Chanel G., Cowley B., Ekman I., Salminen M., Järvelä S., Ravaja N. A review of the use of psychophysiological methods in game research. J. Gaming Virtual Worlds. 2011;3:181–199. doi: 10.1386/jgvw.3.3.181_1. DOI

Drachen A., Nacke L.E., Yannakakis G., Pedersen A.L. Psychophysiological Correlations with Gameplay Experience Dimensions. [(accessed on 25 February 2019)]; Available online: https://arxiv.org/abs/1004.0243.

IJsselsteijn W., Van Den Hoogen W., Klimmt C., De Kort Y., Lindley C., Mathiak K., Poels K., Ravaja N., Turpeinen M., Vorderer P. Measuring the experience of digital game enjoyment; Proceedings of the Measuring Behavior, Noldus Information Tecnology; Wageningen, The Netherlands. 26–29 Augest 2008; pp. 88–89.

Drachen A., Nacke L.E., Yannakakis G., Pedersen A.L. Correlation between heart rate, electrodermal activity and player experience in first-person shooter games; Proceedings of the 5th ACM SIGGRAPH Symposium on Video Games; Los Angeles, CA, USA. 28–29 July 2010; pp. 49–54.

Ballard M.E., Wiest J.R. Mortal Kombat (tm): The Effects of Violent Videogame Play on Males’ Hostility and Cardiovascular Responding1. J. Appl. Soc. Psychol. 1996;26:717–730. doi: 10.1111/j.1559-1816.1996.tb02740.x. DOI

Codispoti M., Surcinelli P., Baldaro B. Watching emotional movies: Affective reactions and gender differences. Int. J. Psychophysiol. 2008;69:90–95. doi: 10.1016/j.ijpsycho.2008.03.004. PubMed DOI

Mandryk R.L., Inkpen K.M., Calvert T.W. Using psychophysiological techniques to measure user experience with entertainment technologies. Behav. Inf. Technol. 2006;25:141–158. doi: 10.1080/01449290500331156. DOI

Yannakakis G.N., Hallam J., Lund H.H. Entertainment capture through heart rate activity in physical interactive playgrounds. User Model. User-Adapted Interact. 2008;18:207–243. doi: 10.1007/s11257-007-9036-7. DOI

Pedersen C., Togelius J., Yannakakis G.N. Modeling player experience in super mario bros; Proceedings of the IEEE Symposium on Computational Intelligence and Games; Milano, Italy. 7–10 September 2009; pp. 132–139.

Drachen A., Canossa A., Yannakakis G.N. Player modeling using self-organization in Tomb Raider: Underworld; Proceedings of the IEEE Symposium on Computational Intelligence and Games; Milano, Italy. 7–10 September 2009; pp. 1–8.

Togelius J., De Nardi R., Lucas S.M. Making Racing Fun through Player Modeling and Track Evolution. [(accessed on 15 March 2018)]; Available online: http://cogprints.org/5221/1/Togelius2006Making.pdf.

Lobel A., Gotsis M., Reynolds E., Annetta M., Engels R.C., Granic I. Designing and utilizing biofeedback games for emotion regulation: The case of nevermind; Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems; San Jose, CA, USA. 7–12 May 2016; pp. 1945–1951.

Missura O., Gärtner T. Player modeling for intelligent difficulty adjustment; Proceedings of the International Conference on Discovery Science; Porto, Portugal. 3–5 October 2009; pp. 197–211.

Latha C.P., Priya M. A Review on Deep Learning Algorithms for Speech and Facial Emotion Recognition. APTIKOM J. Comput. Sci. Inf. Technol. 2016;1:88–104.

Chen L.S.H., Huang T.S. Joint Processing of Audio-Visual Information for the Recognition of Emotional Expressions in Human-Computer Interaction. [(accessed on 25 February 2019)]; Available online: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.76.1185&rep=rep1&type=pdf.

Diaz R.L., Wong U., Hodgins D.C., Chiu C.G., Goghari V.M. Violent video game players and non-players differ on facial emotion recognition. Aggress. Behav. 2016;42:16–28. doi: 10.1002/ab.21602. PubMed DOI

Martinez H.P., Bengio Y., Yannakakis G.N. Learning deep physiological models of affect. IEEE Comput. Intell. Mag. 2013;8:20–33. doi: 10.1109/MCI.2013.2247823. DOI

Zillmann D. Mechanisms of emotional involvement with drama. Poetics. 1995;23:33–51. doi: 10.1016/0304-422X(94)00020-7. DOI

Sherry J.L. Flow and media enjoyment. Commun. Theory. 2004;14:328–347. doi: 10.1111/j.1468-2885.2004.tb00318.x. DOI

Hainey T., Connolly T., Stansfield M., Boyle E. The differences in motivations of online game players and offline game players: A combined analysis of three studies at higher education level. Comput. Educ. 2011;57:2197–2211. doi: 10.1016/j.compedu.2011.06.001. DOI

Infantolino Z., Miller G.A. Psychophysiological Methods in Neuroscience. [(accessed on 15 March 2016)]; Available online: http://nobaproject.com/modules/psychophysiological-methods-in-neuroscience.

Tortora G., Derrickson B. Principles of Anatomy and Physiology: Organisation, Support and Movement and Control Systems of the Human Body. John Wiley and Sons; Hoboken, NJ, USA: 2009.

Parasympathetic Nervous System Rest and Digest. [(accessed on 15 March 2016)]; Available online: http://ktmc.info/parasympatheticnervous-system-rest-and-digest.

Kreibig S.D. Autonomic nervous system activity in emotion: A review. Biol. Psychol. 2010;84:394–421. doi: 10.1016/j.biopsycho.2010.03.010. PubMed DOI

Nepi D., Sbrollini A., Agostinelli A., Maranesi E., Morettini M., Di Nardo F., Fioretti S., Pierleoni P., Pernini L., Valenti S., et al. Validation of the heart-rate signal provided by the Zephyr bioharness 3.0; Proceedings of the 2016 IEEE Computing in Cardiology Conference (CinC); Vancouver, BC, Canada. 11–14 September 2016; pp. 361–364.

Stern R.M., Ray W.J., Quigley K.S. Psychophysiological Recording. Oxford University Press; Cary, NC, USA: 2001.

Dawson M.E., Schell A.M., Filion D.L. Principles of Psychophysiology: Physical, Social and Inferential Elements. Cambridge University Press; New York, NY, USA: 2007. The electrodermal system; pp. 295–324.

McCarthy J., Wright P. Technology as experience. Interactions. 2004;11:42–43. doi: 10.1145/1015530.1015549. DOI

Ward R.D., Marsden P.H. Physiological responses to different web page designs. Int. J. Hum. Comput. Stud. 2003;59:199–212. doi: 10.1016/S1071-5819(03)00019-3. DOI

Davidson R.J., Jackson D.C., Larson C.L. Human electroencephalography. Handb. Psychophysiol. 2000;2:27–52.

Smith M.E., McEvoy L.K., Gevins A. Neurophysiological indices of strategy development and skill acquisition. Cogn. Brain Res. 1999;7:389–404. doi: 10.1016/S0926-6410(98)00043-3. PubMed DOI

Aftanas L., Reva N., Varlamov A., Pavlov S., Makhnev V. Analysis of evoked EEG synchronization and desynchronization in conditions of emotional activation in humans: Temporal and topographic characteristics. Neurosci. Behav. Physiol. 2004;34:859–867. doi: 10.1023/B:NEAB.0000038139.39812.eb. PubMed DOI

Certicky M. Correlation between user experience in electronic entertainment and psychophysiological measurements; Proceedings of the Scientific Conference of Young Researchers; Herlany, Slovakia. 30–31 May 2016; pp. 80–83.

Ganglbauer E., Schrammel J., Deutsch S., Tscheligi M. Applying Psychophysiological Methods for Measuring User Experience: Possibilities, Challenges and Feasibility. [(accessed on 25 February 2019)]; Available online: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.189.3410&rep=rep1&type=pdf.

Navalyal G.U., Gavas R.D. A dynamic attention assessment and enhancement tool using computer graphics. Hum.-Centr. Comput. Inf. Sci. 2014;4:11. doi: 10.1186/s13673-014-0011-0. DOI

Hinton G., Deng L., Yu D., Dahl G.E., Mohamed A.R., Jaitly N., Senior A., Vanhoucke V., Nguyen P., Sainath T.N., et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Process. Mag. 2012;29:82–97. doi: 10.1109/MSP.2012.2205597. DOI

Kingma D.P., Ba J. Adam: A method for stochastic optimization. arXiv. 2014. 1412.6980

Criminisi A., Shotton J., Konukoglu E. Decision forests: A unified framework for classification, regression, density estimation, manifold learning and semi-supervised learning. Found. Trends Comput. Graph. Vis. 2012;7:81–227. doi: 10.1561/0600000035. DOI

Shotton J., Sharp T., Kohli P., Nowozin S., Winn J., Criminisi A. Advances in Neural Information Processing Systems. Curran Associates, Inc.; Red Hook, NY, USA: 2013. Decision jungles: Compact and rich models for classification; pp. 234–242.