Remote continuous patient monitoring is an essential feature of eHealth systems, offering opportunities for personalized care. Among its emerging applications, emotion and stress recognition hold significant promise, but face major challenges due to the subjective nature of emotions and the complexity of collecting and interpreting related data. This paper presents a review of open access multimodal datasets used in emotion and stress detection. It focuses on dataset characteristics, acquisition methods, and classification challenges, with attention to physiological signals captured by wearable devices, as well as advanced processing methods of these data. The findings show notable advances in data collection and algorithm development, but limitations remain, e.g., variability in real-world conditions, individual differences in emotional responses, and difficulties in objectively validating emotional states. The inclusion of self-reported and contextual data can enhance model performance, yet lacks consistency and reliability. Further barriers include privacy concerns, annotation of long-term data, and ensuring robustness in uncontrolled environments. By analyzing the current landscape and highlighting key gaps, this study contributes a foundation for future work in emotion recognition. Progress in the field will require privacy-preserving data strategies and interdisciplinary collaboration to develop reliable, scalable systems. These advances can enable broader adoption of emotion-aware technologies in eHealth and beyond.

Department of Telecommunications Brno University of Technology Brno Czechia

Tampere University Hospital Tampere Finland

Wireless Research Center Tampere University Tampere Finland

Zobrazit více v PubMed

Wu Y, Daoudi M, Amad A (2023) Transformer-based self-supervised multimodal representation learning for wearable emotion recognition. IEEE Trans Affect Comput 15(1):157–172 DOI

Mentis A-FA, Lee D, Roussos P (2024) Applications of artificial intelligence - machine learning for detection of stress: a critical overview. Mol Psychiatry 29(6):1882–1894 PubMed DOI

Ayata D, Yaslan Y, Kamasak ME (2020) Emotion recognition from multimodal physiological signals for emotion aware healthcare systems. J Med Biol Eng 40:149–157 DOI

Azam N, Ahmad T, Haq NU (2021) Automatic emotion recognition in healthcare data using supervised machine learning. PeerJ Comput Sci 7:751 PubMed DOI PMC

Gomes N, Pato M, Lourenco AR, Datia N (2023) A survey on wearable sensors for mental health monitoring. Sensors 23(3):1330 PubMed DOI PMC

Robinson T, Condell J, Ramsey E, Leavey G (2023) Self-management of subclinical common mental health disorders (anxiety, depression and sleep disorders) using wearable devices. Int J Environ Res Publ Health 20(3):2636 PubMed DOI PMC

Guo R, Guo H, Wang L, Chen M, Yang D, Li B (2024) Development and application of emotion recognition technology - a systematic literature review. BMC Psychol 12(95):1–15 PubMed PMC

Dunn J, Runge R, Snyder M (2018) Wearables and the medical revolution. Pers Med 15(5):429–448 PubMed DOI PMC

Pereira R, Mendes C, Ribeiro J, Ribeiro R, Miragaia R, Rodrigues N, Costa N, Pereira A (2024) Systematic review of emotion detection with computer vision and deep learning. Sensors 24(11):3484 PubMed DOI PMC

Patel V, Chesmore A, Legner CM, Pandey S (2022) Trends in workplace wearable technologies and connected-worker solutions for next-generation occupational safety, health, and productivity. Adv Intell Syst 4(1):2100099 DOI

Svertoka E, Saafi S, Rusu-Casandra A, Burget R, Marghescu I, Hosek J, Ometov A (2021) Wearables for industrial work safety: a survey. Sensors 21(11):3844 PubMed DOI PMC

De Witte NA, Carlbring P, Etzelmueller A, Nordgreen T, Karekla M, Haddouk L, Belmont A, Øverland S, Abi-Habib R, Bernaerts S et al (2021) Online consultations in mental healthcare during the COVID-19 outbreak: an international survey study on professionals’ motivations and perceived barriers. Internet Interv 25:100405 PubMed DOI PMC

Dhuheir M, Albaseer A, Baccour E, Erbad A, Abdallah M, Hamdi M (2021) Emotion recognition for healthcare surveillance systems using neural networks: a survey. In: Proc. of International Wireless Communications and Mobile Computing (IWCMC), pp 681–687. IEEE

Koelstra S, Muhl C, Soleymani M, Lee J-S, Yazdani A, Ebrahimi T, Pun T, Nijholt A, Patras I (2012) DEAP: a database for emotion analysis; using physiological signals. IEEE Trans Affect Comput 3(1):18–31 DOI

Site A, Lohan ES, Jolanki O, Valkama O, Hernandez RR, Latikka R, Alekseeva D, Vasudevan S, Afolaranmi S, Ometov A et al (2022) Managing perceived loneliness and social-isolation levels for older adults: a survey with focus on wearables-based solutions. Sensors 22(3):1108 PubMed DOI PMC

Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. Proceedings of the IEEE computer society conference on computer vision and pattern recognition workshops, pp 94–101

Saganowski S, Perz B, Polak AG, Kazienko P (2022) Emotion recognition for everyday life using physiological signals from wearables: a systematic literature review. IEEE Trans Affect Comput 14(3):1876–1897 DOI

Berkemeier L, Kamphuis W, Brouwer A-M, De Vries H, Schadd M, Van Baardewijk JU, Oldenhuis H, Verdaasdonk R, Gemert-Pijnen L (2024) Measuring affective state: subject-dependent and-independent prediction based on longitudinal multimodal sensing. IEEE Trans Affect Comput

Booth BM, Vrzakova H, Mattingly SM, Martinez GJ, Faust L, D’Mello SK (2022) Toward robust stress prediction in the age of wearables: modeling perceived stress in a longitudinal study with information workers. IEEE Trans Affect Comput 13(4):2201–2217 DOI

Opoku Asare K, Terhorst Y, Vega J, Peltonen E, Lagerspetz E, Ferreira D (2021) Predicting depression from smartphone behavioral markers using machine learning methods, hyperparameter optimization, and feature importance analysis: exploratory study. JMIR mHealth uHealth 9(7):26540 PubMed DOI PMC

Khare SK, Blanes-Vidal V, Nadimi ES, Acharya UR (2024) Emotion recognition and artificial intelligence: a systematic review (2014–2023) and research recommendations. Inf Fusion 102:102019 DOI

Lin H-C, Ometov A, Arponen O, Nikunen K, Nurmi J (2024) Stress DeTech-tion: revolutionizing wellbeing in future networks. In: Proc. of international conference on information technology for social good, pp 114–117

Arora A, Alderman JE, Palmer J, Ganapathi S, Laws E, Mccradden MD, Oakden-Rayner L, Pfohl SR, Ghassemi M, McKay F et al (2023) The value of standards for health datasets in artificial intelligence-based applications. Nat Med 29(11):2929–2938 PubMed DOI PMC

Zhao S, Jia G, Yang J, Ding G, Keutzer K (2021) Emotion recognition from multiple modalities: fundamentals and methodologies. IEEE Signal Process Mag 38(6):59–73 DOI

D’Mello S, Kory J (2015) A review of affective computing: from unimodal analysis to multimodal fusion. IEEE Trans Affect Comput 7(1):18–31

Zhu X, Guo C, Feng H, Huang Y, Feng Y, Wang X, Wang R (2024) A review of key technologies for emotion analysis using multimodal information. Cogn Comput 16(4):1504–1530 DOI

European Commission (2021) Proposal for a regulation of the European parliament and of the council laying down harmonised rules on artificial intelligence (artificial intelligence act). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:52021PC0206. Accessed 14 May 2025

European Commission (2021) Artificial intelligence act - article 52: transparency obligations. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:52021PC0206. Accessed 14 May 2025

European Union (2016) General data protection regulation – article 9: processing of special categories of personal data. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32016R0679. Accessed 14 May 2025

European Union (2016) General Data Protection Regulation (GDPR). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32016R0679. Accessed 14 May 2025

Vinkers CH, Kuzminskaite E, Lamers F, Giltay EJ, Penninx BW (2021) An integrated approach to understand biological stress system dysregulation across depressive and anxiety disorders. J Affect Disor 283:139–146 PubMed DOI

Giannakakis G, Grigoriadis D, Giannakaki K, Simantiraki O, Roniotis A, Tsiknakis M (2019) Review on psychological stress detection using biosignals. IEEE Trans Affect Comput 13(1):440–460 DOI

Thakur A, Sharma A, Singh A (2020) Predicting mental health using smartphone usage and sensor data. J Ambient Intell Hum Comput 14:17–29

Tervonen J, Puttonen S, Sillanpää MJ, Hopsu L, Homorodi Z, Keränen J, Pajukanta J, Tolonen A, Lämsä A, Mäntyjärvi J (2020) Personalized mental stress detection with self-organizing map: from laboratory to the field. Comput Biol Med 124:103935 PubMed DOI

Sanchez W, Martinez A, Hernandez Y, Estrada H, Gonzalez-Mendoza M (2018) A predictive model for stress recognition in desk jobs. J Ambient Intell Hum Comput 14:17–29 DOI

Sağbaş EA, Korukoglu S, Ballı S (2024) Real-Time stress detection from smartphone sensor data using genetic algorithm-based feature subset optimization and K-nearest neighbor algorithm. Multimed Tools Appl 83(1):1–32 DOI

Can YS, André E (2023) Performance exploration of RNN variants for recognizing daily life stress levels by using multimodal physiological signals. In: Proceedings of the 25th international conference on multimodal interaction, pp 481–487

Sah RK, Cleveland MJ, Ghasemzadeh H (2023) Stress monitoring in free-living environments. IEEE J Biomed Health Inform PubMed

Campanella S, Altaleb A, Belli A, Pierleoni P, Palma L (2024) PPG and EDA dataset collected with Empatica E4 for stress assessment. Data Brief 53:110102 PubMed DOI PMC

Rahmani MH, Symons M, Sobhani O, Berkvens R, Weyn M (2024) EmoWear: wearable physiological and motion dataset for emotion recognition and context awareness. Sci Data 11(1):648 PubMed DOI PMC

Xu X, Zhang H, Sefidgar Y, Ren Y, Liu X, Seo W, Brown J, Kuehn K, Merrill M, Nurius P et al (2022) GLOBEM dataset: multi-year datasets for longitudinal human behavior modeling generalization. Adv Neural Inf Process Syst 35:24655–24692

Grenier A, Lohan ES, Ometov A, Nurmi J (2023) Towards smarter positioning through analyzing raw GNSS and multi-sensor data from android devices: a dataset and an open-source application. Electronics 12(23):4781 DOI

Lin H-C, Ometov A, Arponen O, Nikunen K, Nurmi J (2024) Towards emotion mapping for multimodal unobtrusive stress monitoring. In: Proceedings of 16th International Congress on Ultra Modern Telecommunications and Control Systems and Workshops (ICUMT). IEEE

Saidcan Y (2024) Mood-aware emotion recognition. https://github.com/ysaidcan/mood_aware_emotion_recog [Accessed May 14, 2025]. Accessed 04 Dec 2024

Diaz KM, Krupka DJ, Chang MJ, Peacock J, Ma Y, Goldsmith J, Schwartz JE, Davidson KW (2015) Fitbit®: an accurate and reliable device for wireless physical activity tracking. Int J Cardiol 185:138 PubMed DOI PMC

Dartmouth College (2024) StudentLife dataset. https://studentlife.cs.dartmouth.edu/dataset.html [Accessed May 14, 2025]. Accessed 04 Dec 2024

University of Siegen (2024) WESAD: a multimodal dataset for wearable emotion recognition. https://ubi29.informatik.uni-siegen.de/usi/data_wesad.html [Accessed May 14, 2025]. Accessed 04 Dec 2024

Kirschbaum C, Pirke K-M, Hellhammer DH (1993) The ‘trier social stress test’-a tool for investigating psychobiological stress responses in a laboratory setting. Neuropsychobiology 28(1–2):76–81 PubMed DOI

Sah RK, McDonell M, Pendry P, Parent S, Ghasemzadeh H, Cleveland MJ (2022) Alcohol and Drug Abuse Research Program (ADARP) dataset. https://zenodo.org/record/6640290. Accessed 04 Dec 2024

Saugbacks (2019) Stress detection dataset. https://drive.google.com/drive/folders/1firMkHeE0vRF5V3J7eYenxSZtbHLiCH5https://drive.google.com/drive/folders/1firMkHeE0vRF5V3J7eYenxSZtbHLiCH5 [Accessed May 14, 2025]. Accessed 04 Dec 2024

UW-EXP (2024) GLOBEM: a framework for global multimodal emotion recognition. https://github.com/UW-EXP/GLOBEM/tree/main. Accessed 04 Dec 2024

Schuurmans AA, De Looff P, Nijhof KS, Rosada C, Scholte RH, Popma A, Otten R (2020) Validity of the Empatica E4 wristband to measure Heart Rate Variability (HRV) parameters: a comparison to Electrocardiography (ECG). J Med Syst 44:1–11 PubMed DOI PMC

Campanella S, Altaleb A, Belli A, Pierleoni P, Palma L (2023) EmpaticaE4Stress dataset. Mendeley data. Accessed 04 Dec 2024. 10.17632/kb42z77m2g.2

Rahmani MH, Symons M, Berkvens R, Weyn M (2023) EmoWear: a wearable physiological and motion dataset for emotion recognition and context awareness. Zenodo. Accessed 04 Dec 2024. 10.5281/zenodo.10407279 PubMed PMC

Wang R, Chen F, Chen Z, Li T, Harari G, Tignor S, Zhou X, Ben-Zeev D, Campbell AT (2014) StudentLife: assessing mental health, academic performance and behavioral trends of college students using smartphones. In: Proceedings of ACM international joint conference on pervasive and ubiquitous computing, pp 3–14

Cai H, Yuan Z, Gao Y, Sun S, Li N, Tian F, Xiao H, Li J, Yang Z, Li X et al (2022) A multi-modal open dataset for mental-disorder analysis. Sci Data 9(1):178 PubMed DOI PMC

Zheng W-L, Lu B-L (2015) Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Trans Auton Ment Dev 7(3):162–175 DOI

Katsigiannis S, Ramzan N (2017) DREAMER: a database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices. IEEE J Biomed Health Inform 22(1):98–107 PubMed DOI

Stojchevska B, Stojanovic J, Milenkovic M, Milenkovic A (2022) Assessing the added value of context during stress detection from wearable data. BMC Med Inform Decis Mak 22(1):1–12 PubMed DOI PMC

Smets E, Rios Velazquez E, Schiavone G, Chakroun I, D’Hondt E, De Raedt W, Cornelis J, Janssens O, Van Hoecke S, Claes S et al (2018) Large-scale wearable data reveal digital phenotypes for daily-life stress detection. NPJ Digi Med 1(1):67 PubMed DOI PMC

Gaballah A, Tiwari A, Narayanan S, Falk TH (2021) Context-aware speech stress detection in hospital workers using Bi-LSTM classifiers. In: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp 8348–8352. IEEE

Schmidt P, Reiss A, Duerichen R, Marberger C, Van Laerhoven K (2018) Introducing WESAD, a multimodal dataset for wearable stress and affect detection. In: Proceedings of the 20th ACM international conference on multimodal interaction, pp 400–408

Sanchez W (2024) LabourCheck. https://www.dropbox.com/sh/zb0xbcdnm92lvrp/AAA4aqDL2f5Vsa24dudcyA2_a?dl=0https://www.dropbox.com/sh/zb0xbcdnm92lvrp/AAA4aqDL2f5Vsa24dudcyA2_a?dl=0 [Accessed May 14, 2025]. Accessed 04 Dec 2024

Sağbaş EA, Korukoglu S, Balli S (2020) Stress detection via keyboard typing behaviors by using smartphone sensors and machine learning techniques. J Med Syst 44:1–12 PubMed DOI

Can YS, Chalabianloo N, Ekiz D, Fernandez-Alvarez J, Riva G, Ersoy C (2020) Personal stress-level clustering and decision-level smoothing to enhance the performance of ambulatory stress detection with smartwatches. IEEE Access 8:38146–38163 DOI

Park CY, Cha N, Kang S, Kim A, Khandoker AH, Hadjileontiadis L, Oh A, Jeong Y, Lee U (2020) K-EmoCon, a multimodal sensor dataset for continuous emotion recognition in naturalistic conversations. Sci Data 7(1):293 PubMed DOI PMC

Shah RV, Grennan G, Zafar-Khan M, Alim F, Dey S, Ramanathan D, Mishra J (2021) Personalized machine learning of depressed mood using wearables. Transl Psychiatry 11(1):1–18 PubMed DOI PMC

Can YS, Gokay D, Kılıç DR, Ekiz D, Chalabianloo N, Ersoy C (2020) How laboratory experiments can be exploited for monitoring stress in the wild: a bridge between laboratory and daily life. Sensors 20(3):838 PubMed DOI PMC

Lin WH, Zheng D, Li G, Chen F (2022) Feasibility of a machine learning-based smartphone application for detecting depression and anxiety in seniors. Front Psychol 13:811517 PubMed DOI PMC

Sah RK, McDonell M, Pendry P, Parent S, Ghasemzadeh H, Cleveland MJ (2022) Adarp: a multi modal dataset for stress and alcohol relapse quantification in real life setting. In: Proceedings of IEEE-EMBS international conference on wearable and implantable Body Sensor Networks (BSN), pp 1–4. IEEE

Campanella S, Altaleb A, Belli A, Pierleoni P, Palma L (2023) A method for stress detection using Empatica E4 bracelet and machine-learning techniques. Sensors 23(7):3565 PubMed DOI PMC

Tutunji R, Kogias N, Kapteijns B, Krentz M, Krause F, Vassena E, Hermans EJ (2023) Detecting prolonged stress in real life using wearable biosensors and ecological momentary assessments: naturalistic experimental study. J Med Internet Res 25:39995 PubMed DOI PMC

Yang P, Liu N, Liu X, Shu Y, Ji W, Ren Z, Sheng J, Yu M, Yi R, Zhang D et al (2024) A multimodal dataset for mixed emotion recognition. Sci Data 11(1):847 PubMed DOI PMC

Lee M-H, Shomanov A, Begim B, Kabidenova Z, Nyssanbay A, Yazici A, Lee S-W (2024) EAV: EEG-audio-video dataset for emotion recognition in conversational contexts. Sci Data 11(1):1026. 10.1038/s41597-024-03838-4 PubMed DOI PMC

Sanches (2024) Dataset for emotion recognition (via Dropbox). https://www.dropbox.com/sh/zb0xbcdnm92lvrp/AAA4aqDL2f5Vsa24dudcyA2_a?dl=0 [Accessed May 14, 2025]. Accessed 04 Dec 2024

Charlton PH, Allen J, Bailón R, Baker S, Behar JA, Chen F, Clifford GD, Clifton DA, Davies HJ, Ding C et al (2023) The 2023 wearable photoplethysmography roadmap. Physiol Meas 44(11):111001 PubMed DOI PMC

Sah RK, Ghasemzadeh H (2021) Stress classification and personalization: getting the most out of the least. http://arxiv.org/abs/2107.05666

Schiweck C, Piette D, Berckmans D, Claes S, Vrieze E (2019) Heart rate and high frequency heart rate variability during stress as biomarker for clinical depression. a systematic review. Psychol Med 49(2):200–211 PubMed

Nath RK, Thapliyal H, Caban-Holt A (2022) Machine learning based stress monitoring in older adults using wearable sensors and cortisol as stress biomarker. J Signal Process Syst 94(6):513–525 DOI

Tazarv A, Labbaf S, Reich SM, Dutt N, Rahmani AM, Levorato M (2021) Personalized stress monitoring using wearable sensors in everyday settings. In: Proceedings of 43rd annual international conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp 7332–7335. IEEE PubMed

Maxhuni A, Hernandez-Leal P, Morales EF, Sucar LE, Osmani V, Mayora O (2020) Unobtrusive stress assessment using smartphones. IEEE Trans Mob Comput 20(6):2313–2325 DOI

Fukazawa Y, Ito T, Okimura T, Yamashita Y, Maeda T, Ota J (2019) Predicting anxiety state using smartphone-based passive sensing. J Biomed Inform 93:103151 PubMed DOI

Kaczor EE, Carreiro S, Stapp J, Chapman B, Indic P (2020) Objective measurement of physician stress in the emergency department using a wearable sensor. In: Proceedings of the annual Hawaii international conference on system sciences, vol 2020, p 3729. NIH Public Access PubMed PMC

Thieme A, Belgrave D, Doherty G (2020) Machine learning in mental health: a systematic review of the HCI literature to support the development of effective and implementable ML systems. ACM Trans Comput-Hum Interaction (TOCHI) 27(5):1–53 DOI

Ghods A, Shahrokni A, Ghasemzadeh H, Cook D (2021) Remote monitoring of the performance status and burden of symptoms of patients with gastrointestinal cancer via a consumer-based activity tracker: quantitative cohort study. JMIR Cancer 7(4):22931 PubMed DOI PMC

Harati S, Sharma A, Singh A (2020) Detecting depression using multimodal data from smartphones. J Ambient Intell Hum Comput 14:17–29

Sun Y, Zhang W, Liu X, Wang Y (2023) Clustering-fusion feature selection method for identifying major depressive disorder using resting-state EEG signals. IEEE J Biomed Health Inform 27(4):1813–1824 PubMed

Chikersal P, Doryab A, Tumminia M, Villalba DK, Dutcher JM, Liu X, Cohen S, Creswell KG, Mankoff J, Creswell JD et al (2021) Detecting depression and predicting its onset using longitudinal symptoms captured by passive sensing: a machine learning approach with robust feature selection. ACM Trans Comput-Hum Interaction (TOCHI) 28(1):1–41 DOI

Zhao R, Liu T, Huang Z, Lun DP, Lam KM (2021) Deep learning model for emotion recognition using EEG signals. J Ambient Intell Hum Comput 14:17–29

Sano A, Taylor S, McHill AW, Phillips AJ, Barger LK, Klerman E, Picard R (2018) Identifying objective physiological markers and modifiable behaviors for self-reported stress and mental health status using wearable sensors and mobile phones: observational study. J Med Internet Res 20(6):210 PubMed DOI PMC