This paper presents a gamified motor imagery brain-computer interface (MI-BCI) training in immersive virtual reality. The aim of the proposed training method is to increase engagement, attention, and motivation in co-adaptive event-driven MI-BCI training. This was achieved using gamification, progressive increase of the training pace, and virtual reality design reinforcing body ownership transfer (embodiment) into the avatar. From the 20 healthy participants performing 6 runs of 2-class MI-BCI training (left/right hand), 19 were trained for a basic level of MI-BCI operation, with average peak accuracy in the session = 75.84%. This confirms the proposed training method succeeded in improvement of the MI-BCI skills; moreover, participants were leaving the session in high positive affect. Although the performance was not directly correlated to the degree of embodiment, subjective magnitude of the body ownership transfer illusion correlated with the ability to modulate the sensorimotor rhythm.

Faculty of Informatics Masaryk University Brno Czechia

Zobrazit více v PubMed

Ahn M., Jun S. C. (2015). Performance variation in motor imagery brain–computer interface: a brief review. J. Neurosci. Methods 243, 103–110. 10.1016/j.jneumeth.2015.01.033 PubMed DOI

Ahn S., Jun S. C. (2012). Feasibility of hybrid BCI using ERD- and SSSEP- BCI, in 2012 12th International Conference on Control, Automation and Systems (JeJu Island: ), 2053–2056.

Alimardani M., Nishio S., Ishiguro H. (2013). Humanlike robot hands controlled by brain activity arouse illusion of ownership in operators. Sci. Rep. 3:2396. 10.1038/srep02396 PubMed DOI PMC

Alimardani M., Nishio S., Ishiguro H. (2016a). The importance of visual feedback design in BCIs; from embodiment to motor imagery learning. PLoS ONE 11:e0161945. 10.1371/journal.pone.0161945 PubMed DOI PMC

Alimardani M., Nishio S., Ishiguro H. (2016b). Removal of proprioception by BCI raises a stronger body ownership illusion in control of a humanlike robot. Sci. Rep. 6:33514. 10.1038/srep33514 PubMed DOI PMC

Alimardani M., Nishio S., Ishiguro H. (2018). Exploring minimal requirement for body ownership transfer by brain–computer interface, in Geminoid Studies: Science and Technologies for Humanlike Teleoperated Androids, eds Ishiguro H., Dalla Libera F. (Singapore: Springer; ), 329–338.

Barsotti M., Leonardis D., Vanello N., Bergamasco M., Frisoli A. (2018). Effects of continuous kinaesthetic feedback based on tendon vibration on motor imagery BCI performance. IEEE Trans. Neural Syst. Rehabil. Eng. 26, 105–114. 10.1109/TNSRE.2017.2739244 PubMed DOI

Bell C. J., Shenoy P., Chalodhorn R., Rao R. P. N. (2008). Control of a humanoid robot by a noninvasive brain–computer interface in humans. J. Neural Eng. 5, 214–220. 10.1088/1741-2560/5/2/012 PubMed DOI

Blanke O., Metzinger T. (2009). Full-body illusions and minimal phenomenal selfhood. Trends Cogn. Sci. 13, 7–13. 10.1016/j.tics.2008.10.003 PubMed DOI

Botvinick M., Cohen J. (1998). Rubber hands' feel'touch that eyes see. Nature 391:756. 10.1038/35784 PubMed DOI

Braun N., Emkes R., Thorne J. D., Debener S. (2016). Embodied neurofeedback with an anthropomorphic robotic hand. Sci. Rep. 6:37696. 10.1038/srep37696 PubMed DOI PMC

Cao T., Wan F., Wong C. M., da Cruz J. N., Hu Y. (2014). Objective evaluation of fatigue by EEG spectral analysis in steady-state visual evoked potential-based brain-computer interfaces. Biomed. Eng. Online 13:28. 10.1186/1475-925X-13-28 PubMed DOI PMC

Chang C., Hsu S., Pion-Tonachini L., Jung T. (2018). Evaluation of artifact subspace reconstruction for automatic EEG artifact removal, in 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (Honolulu, HI: ), 1242–1245. PubMed

Chavarriaga R., Fried-Oken M., Kleih S., Lotte F., Scherer R. (2017). Heading for new shores! Overcoming pitfalls in BCI design. Brain-Comput. Interfaces 4, 60–73. 10.1080/2326263X.2016.1263916 PubMed DOI PMC

Cohen O., Druon S., Lengagne S., Mendelsohn A., Malach R., Kheddar A., et al. (2012). fMRI robotic embodiment: a pilot study, in 2012 4th IEEE RAS EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob) (Rome: ), 314–319.

David N., Newen A., Vogeley K. (2008). The “sense of agency” and its underlying cognitive and neural mechanisms. Conscious. Cogn. 17, 523–534. 10.1016/j.concog.2008.03.004 PubMed DOI

de Freitas S. (2011). Technology: game for change. Nature 470, 330–331. 10.1038/470330a DOI

Delorme A., Makeig S. (2004). EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J. Neurosci. Methods 134, 9–21. 10.1016/j.jneumeth.2003.10.009 PubMed DOI

Dickhaus T., Sannelli C., Müller K.-R., Curio G., Blankertz B. (2009). Predicting BCI performance to study BCI illiteracy. BMC Neurosci. 10(Suppl. 1):P84 10.1186/1471-2202-10-S1-P84 DOI

Dummer T., Picot-Annand A., Neal T., Moore C. (2009). Movement and the rubber hand illusion. Perception 38, 271–280. 10.1068/p5921 PubMed DOI

Ehrsson H. H. (2012). The concept of body ownership and its relation to multisensory integration, in The New Handbook of Multisensory Process, ed Stein B. (Cambridge, MA: MIT Press; ), 19.

Ehrsson H. H., Holmes N. P., Passingham R. E. (2005). Touching a rubber hand: feeling of body ownership is associated with activity in multisensory brain areas. J. Neurosci. 25, 10564–10573. 10.1523/JNEUROSCI.0800-05.2005 PubMed DOI PMC

Evans N., Gale S., Schurger A., Blanke O. (2015). Visual feedback dominates the sense of agency for brain-machine actions. PLoS ONE 10:e0130019. 10.1371/journal.pone.0130019 PubMed DOI PMC

Gallagher S. (2000). Philosophical conceptions of the self: implications for cognitive science. Trends Cogn. Sci. 4, 14–21. 10.1016/S1364-6613(99)01417-5 PubMed DOI

Gallagher S. (2007). The natural philosophy of agency. Philos. Compass 2, 347–357. 10.1111/j.1747-9991.2007.00067.x DOI

Grosse-Wentrup M., Schölkopf B. (2012). High gamma-power predicts performance in sensorimotor-rhythm brain–computer interfaces. J. Neural Eng. 9:046001. 10.1088/1741-2560/9/4/046001 PubMed DOI

Guger C., Allison B. Z., Grosswindhager B., Prückl R., Hintermüller C., Kapeller C., et al. . (2012). How many people could use an SSVEP BCI? Front. Neurosci. 6:169. 10.3389/fnins.2012.00169 PubMed DOI PMC

Hamari J., Koivisto J., Sarsa H. (2014). Does gamification work? – A literature review of empirical studies on gamification, in 2014 47th Hawaii International Conference on System Sciences (Waikoloa, HI: ), 3025–3034. 10.1109/HICSS.2014.377 DOI

Hammer E. M., Halder S., Blankertz B., Sannelli C., Dickhaus T., Kleih S., et al. . (2012). Psychological predictors of SMR-BCI performance. Biol. Psychol. 89, 80–86. 10.1016/j.biopsycho.2011.09.006 PubMed DOI

Hwang H.-J., Kwon K., Im C.-H. (2009). Neurofeedback-based motor imagery training for brain–computer interface (BCI). J. Neurosci. Methods 179, 150–156. 10.1016/j.jneumeth.2009.01.015 PubMed DOI

Jeannerod M. (2007). Being oneself. J. Physiol. Paris 101, 161–168. 10.1016/j.jphysparis.2007.11.005 PubMed DOI

Jeunet C., Jahanpour E., Lotte F. (2016a). Why standard brain-computer interface (BCI) training protocols should be changed: an experimental study. J. Neural Eng. 13:036024. 10.1088/1741-2560/13/3/036024 PubMed DOI

Jeunet C., N'Kaoua B., Lotte F. (2016b). Advances in user-training for mental-imagery-based BCI control: psychological and cognitive factors and their neural correlates. Prog. Brain Res. 228, 3–35. 10.1016/bs.pbr.2016.04.002 PubMed DOI

Juliano J. M., Spicer R. P., Vourvopoulos A., Lefebvre S., Jann K., Ard T., et al. (2019). Embodiment is related to better performance on an immersive brain computer interface in head-mounted virtual reality: a pilot study. bioRxiv 578682. 10.1101/578682 PubMed DOI PMC

Kaiser V., Bauernfeind G., Kreilinger A., Kaufmann T., Kübler A., Neuper C., et al. . (2014). Cortical effects of user training in a motor imagery based brain–computer interface measured by fNIRS and EEG. NeuroImage 85, 432–444. 10.1016/j.neuroimage.2013.04.097 PubMed DOI

Kalckert A., Ehrsson H. H. (2012). Moving a rubber hand that feels like your own: a dissociation of ownership and agency. Front. Hum. Neurosci. 6:40. 10.3389/fnhum.2012.00040 PubMed DOI PMC

Kilteni K., Groten R., Slater M. (2012). The sense of embodiment in virtual reality. Presence 21, 373–387. 10.1162/PRES_a_00124 DOI

Kishore S., González-Franco M., Hintemüller C., Kapeller C., Guger C., Slater M., et al. (2014). Comparison of SSVEP BCI and eye tracking for controlling a humanoid robot in a social environment. Presence 23, 242–252. 10.1162/PRES_a_00192 DOI

Kokkinara E., Kilteni K., Blom K. J., Slater M. (2016). First person perspective of seated participants over a walking virtual body leads to illusory agency over the walking. Sci. Rep. 6:28879. 10.1038/srep28879 PubMed DOI PMC

Kosmyna N., Lécuyer A. (2017). Designing guiding systems for brain-computer interfaces. Front. Hum. Neurosci. 11:396. 10.3389/fnhum.2017.00396 PubMed DOI PMC

Krausz G., Scherer R., Korisek G., Pfurtscheller G. (2003). Critical decision-speed and information transfer in the “Graz Brain–Computer Interface. Appl. Psychophysiol. Biofeedback 28, 233–240. 10.1023/A:1024637331493 PubMed DOI

Kübler A., Neumann N., Wilhelm B., Hinterberger T., Birbaumer N. (2004). Predictability of brain-computer communication. J. Psychophysiol. 18, 121–129. 10.1027/0269-8803.18.23.121 DOI

Leeb R., Lee F., Keinrath C., Scherer R., Bischof H., Pfurtscheller G. (2007). Brain-computer communication: motivation, aim, and impact of exploring a virtual apartment. IEEE Trans. Neural Syst. Rehabil. Eng. 15, 473–482. 10.1109/TNSRE.2007.906956 PubMed DOI

Leonardis D., Frisoli A., Solazzi M., Bergamasco M. (2012). Illusory perception of arm movement induced by visuo-proprioceptive sensory stimulation and controlled by motor imagery, in 2012 IEEE Haptics Symposium (HAPTICS) (Vancouver, BC: ), 421–424. 10.1109/HAPTIC.2012.6183825 DOI

Longo M. R., Schüür F., Kammers M. P. M., Tsakiris M., Haggard P. (2008). What is embodiment? A psychometric approach. Cognition 107, 978–998. 10.1016/j.cognition.2007.12.004 PubMed DOI

Lopez S., Bini F., Del Percio C., Marinozzi F., Celletti C., Suppa A., et al. . (2017). Electroencephalographic sensorimotor rhythms are modulated in the acute phase following focal vibration in healthy subjects. Neuroscience 352, 236–248. 10.1016/j.neuroscience.2017.03.015 PubMed DOI

Lotte F., Bougrain L., Clerc M. (2015). Electroencephalography (EEG)-based brain-computer interfaces, in Wiley Encyclopedia of Electrical and Electronics Engineering, ed. Webster J. G. (New York, NY: Wiley-Interscience; ). Available online at: https://onlinelibrary.wiley.com/action/showCitFormats?doi=10.1002%2F047134608X.W8278

Lotte F., Larrue F., Muehl C. (2013). Flaws in current human training protocols for spontaneous Brain-Computer Interfaces: lessons learned from instructional design. Front. Hum. Neurosci. 7:568. 10.3389/fnhum.2013.00568 PubMed DOI PMC

Martens N., Jenke R., Abu-Alqumsan M., Kapeller C., Hintermüller C., Guger C., et al. (2012). Towards robotic re-embodiment using a Brain-and-Body-Computer Interface, in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems (Vilamoura: ), 5131–5132.

Müller-Putz G. R., Scherer R., Brunner C., Leeb R., Pfurtscheller G. (2008). Better than random? A closer look on BCI results. Int. J. Bioelectromagn. 10, 52–55.

Neuper C., Scherer R., Wriessnegger S., Pfurtscheller G. (2009). Motor imagery and action observation: modulation of sensorimotor brain rhythms during mental control of a brain-computer interface. Clin. Neurophysiol. 120, 239–247. 10.1016/j.clinph.2008.11.015 PubMed DOI

Nijboer F., Birbaumer N., Kubler A. (2010). The influence of psychological state and motivation on brain—computer interface performance in patients with amyotrophic lateral sclerosis - a longitudinal study. Front. Neurosci. 4:55. 10.3389/fnins.2010.00055 PubMed DOI PMC

Nijboer F., Furdea A., Gunst I., Mellinger J., McFarland D. J., Birbaumer N., et al. . (2008). An auditory brain-computer interface (BCI). J. Neurosci. Methods 167, 43–50. 10.1016/j.jneumeth.2007.02.009 PubMed DOI PMC

Oculus (2013). Oculus VR – Latency Tester Demo. Available online at: https://www.youtube.com/watch?v=z4y4Nfj-MlM (accessed July 10, 2019).

Penaloza C. I., Alimardani M., Nishio S. (2018). Android feedback-based training modulates sensorimotor rhythms during motor imagery. IEEE Trans. Neural Syst. Rehabil. Eng. 26, 666–674. 10.1109/TNSRE.2018.2792481 PubMed DOI

Perez-Marcos D., Slater M., Sanchez-Vives M. V. (2009). Inducing a virtual hand ownership illusion through a brain-computer interface. Neuroreport 20, 589–594. 10.1097/WNR.0b013e32832a0a2a PubMed DOI

Petit D., Gergondet P., Cherubini A., Kheddar A. (2015). An integrated framework for humanoid embodiment with a BCI, in 2015 IEEE International Conference on Robotics and Automation (ICRA) (Seattle, WA: ), 2882–2887.

Pfurtscheller G., Lopes da Silva F. H. (1999). Event-related EEG/MEG synchronization and desynchronization: basic principles. Clin. Neurophysiol. 110, 1842–1857. 10.1016/S1388-2457(99)00141-8 PubMed DOI

Pfurtscheller G., Neuper C. (2001). Motor imagery and direct brain-computer communication. Proc. IEEE 89, 1123–1134. 10.1109/5.939829 DOI

Raaen K., Kjellmo I. (2015). Measuring latency in virtual reality systems, in Entertainment Computing - ICEC 2015, Vol. 9353, eds Chorianopoulos K., Divitini M., Baalsrud Hauge J., Jaccheri L., Malaka R. (Cham: Springer International Publishing; ), 457–462. 10.1007/978-3-319-24589-8_40 DOI

Renard Y., Lotte F., Gibert G., Congedo M., Maby E., Delannoy V., et al. (2010). Openvibe: an open-source software platform to design, test, and use brain-computer interfaces in real and virtual environments. Presence 19, 35–53. 10.1162/pres.19.1.35 DOI

Shannon C. E. (1949). A mathematical theory of communication, in Bell System Technical Journal, Vol. 27 of 3 (American Telephone and Telegraph Company), 379–423.

Shenoy P., Krauledat M., Blankertz B., Rao R. P. N., Müller K.-R. (2006). Towards adaptive classification for BCI. J. Neural Eng. 3, R13–R23. 10.1088/1741-2560/3/1/R02 PubMed DOI

Škola F., Liarokapis F. (2018). Embodied VR environment facilitates motor imagery brain–computer interface training. Comput. Graph. 75, 59–71. 10.1016/j.cag.2018.05.024 DOI

Slater M., Perez-Marcos D., Ehrsson H. H., Sanchez-Vives M. V. (2009). Inducing illusory ownership of a virtual body. Front. Neurosci. 3, 214–220. 10.3389/neuro.01.029.2009 PubMed DOI PMC

Sollfrank T., Hart D., Goodsell R., Foster J., Tan T. (2015). 3d visualization of movements can amplify motor cortex activation during subsequent motor imagery. Front. Hum. Neurosci. 9:463. 10.3389/fnhum.2015.00463 PubMed DOI PMC

Sollfrank T., Ramsay A., Perdikis S., Williamson J., Murray-Smith R., Leeb R., et al. . (2016). The effect of multimodal and enriched feedback on SMR-BCI performance. Clin. Neurophysiol. 127, 490–498. 10.1016/j.clinph.2015.06.004 PubMed DOI

Sweller J., van Merrienboer J. J. G., Paas F. G. W. C. (1998). Cognitive architecture and instructional design. Educ. Psychol. Rev. 10, 251–296. 10.1023/A:1022193728205 DOI

Talukdar U., Hazarika S. M., Gan J. Q. (2019). Motor imagery and mental fatigue: inter-relationship and EEG based estimation. J. Comput. Neurosci. 46, 55–76. 10.1007/s10827-018-0701-0 PubMed DOI

Tsakiris M., Haggard P., Franck N., Mainy N., Sirigu A. (2005). A specific role for efferent information in self-recognition. Cognition 96, 215–231. 10.1016/j.cognition.2004.08.002 PubMed DOI

Vlek R., van Acken J.-P., Beursken E., Roijendijk L., Haselager P. (2014). BCI and a user's judgment of agency, in Brain-Computer-Interfaces in their ethical, social and cultural contexts, The International Library of Ethics, Law and Technology, eds Grübler G., Hildt E. (Dordrecht: Springer Netherlands; ), 193–202.

Vollmeyer R., Rheinberg F. (1998). Motivationale Einflüsse auf Erwerb und Anwendung von Wissen in einem computersimulierten System. [Motivational influences on the acquisition and application of knowledge in a simulated system.]. Ger. J. Educ. Psychol. 12, 11–23.

von Holst E., Mittelstaedt H. (1950). Das Reafferenzprinzip. Naturwissenschaften 37, 464–476. 10.1007/BF00622503 DOI

Vorderer P., Hartmann T., Klimmt C. (2003). Explaining the enjoyment of playing video games: the role of competition, in Proceedings of the Second International Conference on Entertainment Computing, ICEC '03 (Pittsburgh, PA: Carnegie Mellon University; ), 1–9.

Vourvopoulos A., Ferreira A., Badia S. B. I. (2016). NeuRow: an immersive VR environment for motor-imagery training with the use of brain-computer interfaces and vibrotactile feedback, in 3rd International Conference on Physiological Computing Systems (SCITEPRESS - Science and Technology Publications: ) (Lisbon), 43–53.

Vourvopoulos A., Pardo O. M., Lefebvre S., Neureither M., Saldana D., Jahng E., et al. . (2019). Effects of a brain-computer interface with virtual reality (VR) neurofeedback: a pilot study in chronic stroke patients. Front. Hum. Neurosci. 13:210. 10.3389/fnhum.2019.00210 PubMed DOI PMC

Winkler I., Brandl S., Horn F., Waldburger E., Allefeld C., Tangermann M. (2014). Robust artifactual independent component classification for BCI practitioners. J. Neural Eng. 11:035013. 10.1088/1741-2560/11/3/035013 PubMed DOI

Wolpaw J. R., Birbaumer N., McFarland D. J., Pfurtscheller G., Vaughan T. M. (2002). Brain-computer interfaces for communication and control. Clin. Neurophysiol. 113, 767–791. 10.1016/S1388-2457(02)00057-3 PubMed DOI

Yao L., Meng J., Sheng X., Zhang D., Zhu X. (2014). A novel calibration and task guidance framework for motor imagery BCI via a tendon vibration induced sensation with kinesthesia illusion. J. Neural Eng. 12:016005. 10.1088/1741-2560/12/1/016005 PubMed DOI