The paper is devoted to the study of facial region temperature changes using a simple thermal imaging camera and to the comparison of their time evolution with the pectoral area motion recorded by the MS Kinect depth sensor. The goal of this research is to propose the use of video records as alternative diagnostics of breathing disorders allowing their analysis in the home environment as well. The methods proposed include (i) specific image processing algorithms for detecting facial parts with periodic temperature changes; (ii) computational intelligence tools for analysing the associated videosequences; and (iii) digital filters and spectral estimation tools for processing the depth matrices. Machine learning applied to thermal imaging camera calibration allowed the recognition of its digital information with an accuracy close to 100% for the classification of individual temperature values. The proposed detection of breathing features was used for monitoring of physical activities by the home exercise bike. The results include a decrease of breathing temperature and its frequency after a load, with mean values -0.16 °C/min and -0.72 bpm respectively, for the given set of experiments. The proposed methods verify that thermal and depth cameras can be used as additional tools for multimodal detection of breathing patterns.

Czech Institute of Informatics Robotics and Cybernetics Czech Technical University Prague 166 36 Prague Czech Republic

Department of Computing and Control Engineering University of Chemistry and Technology Prague 166 28 Prague Czech Republic

Faculty of Applied Informatics Tomas Bata University in Zlín 760 05 Zlín Czech Republic

Faculty of Medicine in Hradec Králové Department of Neurology Charles University 500 05 Hradec Kralove Czech Republic

School of Electrical and Electronic Engineering Newcastle University Newcastle upon Tyne NE1 7RU UK

Zobrazit více v PubMed

Procházka A., Schatz M., Vyšata O. Microsoft kinect visual and depth sensors for breathing and heart rate analysis. Sensors. 2016;16:1–11. doi: 10.3390/s16070996. PubMed DOI PMC

Lee J., Hong M., Ryu S. Sleep monitoring system using kinect sensor. Int. J. Distrib. Sens. Netw. 2015;2015 doi: 10.1155/2015/875371. DOI

Procházka A., Vyšata O., Vališ M., Ťupa O., Schatz M., Mařík V. Bayesian classification and analysis of gait disorders using image and depth sensors of Microsoft Kinect. Digit. Signal Prog. 2015;47:169–177. doi: 10.1016/j.dsp.2015.05.011. DOI

Procházka A., Vyšata O., Vališ M., Ťupa O., Schatz M., Mařík V. Use of Image and depth sensors of the Microsoft Kinect for the detection of gait disorders. Neural Comput. Appl. 2015;26:1621–1629. doi: 10.1007/s00521-015-1827-x. DOI

Erden F., Velipasalar S., Alkar A., Cetin A. Sensors in assisted living. IEEE Signal Process. Mag. 2016;33:36–44. doi: 10.1109/MSP.2015.2489978. PubMed DOI

Procházka A., Schätz M., Centonze F., Kuchyňka J., Vyšata O., Vališ M. Extraction of breathing features using MS Kinect for sleep stage detection. Signal Image Video Process. 2016;10:1278–1286. doi: 10.1007/s11760-016-0897-2. DOI

Appel V., Belini V., Jong D., Magalhães D., Caurin G. Classifying emotions in rehabilitation robotics based on facial skin temperature; Proceedings of the IEEE RAS and EMBS International Conference on Biomedical Robotics and Biomechatronics; Sao Paulo, Brazil. 12–14 August 2014; pp. 276–281.

Boccanfuso L., Wang Q., Leite I., Li B., Torres C., Chen L., Salomons N., Foster C., Barney E., Ahn Y., et al. A thermal emotion classifier for improved human–robot interaction; Proceedings of the 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN); New York, NY, USA. 26–31 August 2016; pp. 718–723.

Kwaśniewska A., Rumiński J. Face detection in image sequences using a portable thermal camera; Proceedings of the 13th Quantitative Infrared Thermography Conference; Gdansk, Poland. 4–8 July 2016.

Latif M., Md. Yusof H., Sidek S., Rusli N., Fatai S. Emotion detection from thermal facial imprint based on GLCM features. ARPN-JEAS. 2016;11:345–350.

Nguyen H., Kotani K., Chen F., Le B. Estimation of human emotions using thermal facial information; Proceedings of the SPIE—The International Society for Optical Engineering, ICGIP 2013; Hong Kong, China. 26–27 October 2013.

Rahulamathavan Y., Phan R.C.V., Chambers J.A., Parish D.J. Facial expression recognition in the encrypted domain based on local fisher discriminant analysis. IEEE Trans. Affect. Comput. 2013;4:83–92. doi: 10.1109/T-AFFC.2012.33. DOI

Cheong Y., Yap V., Nisar H. A novel face detection algorithm using thermal imaging; Proceedings of the 2014 IEEE Symposium on Computer Applications and Industrial Electronics, ISCAIE; Penang, Malaysia. 7–8 April 2014; pp. 208–213.

Liu P., Yin L. Spontaneous facial expression analysis based on temperature changes and head motions; Proceedings of the11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, FG 2015; Ljubljana, Slovenia. 4–8 May 2015.

Cardone D., Pinti P., Merla A. Thermal infrared imaging-based computational psychophysiology for psychometrics. Comput. Math. Method Med. 2015;2015:1–8. doi: 10.1155/2015/984353. PubMed DOI PMC

Ioannou S., Gallese V., Merla A. Thermal infrared imaging in psychophysiology: Potentialities and limits. Psychophysiology. 2014;51:951–963. doi: 10.1111/psyp.12243. PubMed DOI PMC

Nhan B., Chau T. Classifying affective states using thermal infrared imaging of the human face. IEEE Trans. Biomed. Eng. 2010;57:979–987. doi: 10.1109/TBME.2009.2035926. PubMed DOI

Hong K., Hong S. Real-time stress assessment using thermal imaging. Vis. Comput. 2016;32:1369–1377. doi: 10.1007/s00371-015-1164-1. DOI

Engert V., Merla A., Grant J., Cardone D., Tusche A., Singer T. Exploring the use of thermal infrared imaging in human stress research. PLoS ONE. 2014;9:e90782. doi: 10.1371/journal.pone.0090782. PubMed DOI PMC

Kim H., Kim J.-Y., Im C.-H. Fast and robust real-time estimation of respiratory rate from photoplethysmography. Sensors. 2016;16:1494. doi: 10.3390/s16091494. PubMed DOI PMC

Zhang X., Ding Q. Respiratory rate estimation from the photoplethysmogram via joint sparse signal reconstruction and spectra Psion. Biomed. Signal Process. Control. 2017;35:1–7. doi: 10.1016/j.bspc.2017.02.003. DOI

Hu M.H., Zhai G.T., Li D., Fan Y.Z., Chen X.H., Yang X.K. Synergetic use of thermal and visible imaging techniques for contactless and unobtrusive breathing measurement. J. Biomed. Opt. 2017;22:1–11. doi: 10.1117/1.JBO.22.3.036006. PubMed DOI

Lin Y.-D., Chien Y.-H., Chen Y.-H. Wavelet-based embedded algorithm for respiratory rate estimation from PPG signal. Biomed. Signal Process. Control. 2017;36:138–145. doi: 10.1016/j.bspc.2017.03.009. DOI

Carpagnano G.E., Foschino-Barbaro M.P., Crocetta C., Lacedonia D., Saliani V., Zoppo L.D., Barnes P.J. Validation of the exhaled breath temperature measure: Reference values in healthy subjects. Chest. 2017;151:855–860. doi: 10.1016/j.chest.2016.11.013. PubMed DOI

Khalidi F.Q., Saatchi R., Burke D., Elphick H., Tan S. Respiration rate monitoring methods: A review. Pediatr. Pulmonol. 2011;46:523–529. doi: 10.1002/ppul.21416. PubMed DOI

Adib F., Mao H., Kabelac Z., Katabi D., Miller R.C. Smart homes that monitor breathing and heart rate; Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI 2015; Seoul, Korea. 18–23 April 2015; pp. 837–846.

Heck D.H., McAfee S.S., Liu Y., Babajani-Feremi A., Rezaie R., Freeman W.J., Wheless J.W., Papanicolaou A.C., Ruszinko M., Sokolov Y., Kozma R. Breathing as a fundamental rhytm of brain function. Front. Neural Circuits. 2017;10:115. doi: 10.3389/fncir.2016.00115. PubMed DOI PMC

Murthy R., Pavlidis I., Tsiamyrtzis P. Touchless Monitoring of breathing function; Proceedings of the 26th Annual International Conference of the IEEE EMBS; San Francisco, CA, USA. 1–5 September 2004. PubMed

Al-Obaisi F., Alqatawna J., Faris H., Rodan A., Al-Kadi O. Pattern recognition of thermal images for monitoring of breathing function. Int. J. Control Autom. 2015;8:381–392. doi: 10.14257/ijca.2015.8.6.37. DOI

Folke M., Cernerud L., Ekström M., Hök B. Critical review of non-invasive respiratory monitoring in medical care. Med. Biol. Eng. Comput. 2003;41:377–383. doi: 10.1007/BF02348078. PubMed DOI

Usamentiaga R., Venegas P., Guerediaga J., Vega L., Molleda J., Bulnes F.G. Infrared Thermography for temperature measurement and non-destructive testing. Sensors. 2014;14:12305–12348. doi: 10.3390/s140712305. PubMed DOI PMC

Xia J., Siochi R.A. A real-time respiratory motion monitoring system using microsoft kinect sensor. Med. Phys. 2012;39:2682–2685. doi: 10.1118/1.4704644. PubMed DOI

Griessenberger H., Heib D.P.J., Kunz A.B., Hoedlmoser K., Schabus M. Assessment of a wireless headband for automatic sleep scoring. Sleep Breath. 2013;17:747–752. doi: 10.1007/s11325-012-0757-4. PubMed DOI PMC

Kolb A., Barth E., Koch R., Larsen R. Time-of-flight sensors in computer graphics. In: Pauly M., Greiner G., editors. Eurographics 2009—State of the Art Reports. The Eurographics Association; Geneva, Switzerland: 2009. pp. 119–134.

Charvátová H., Procházka A., Vaseghi S., Vyšata O., Vališ M. GPS-based analysis of physical activities using positioning and heart rate cycling data. Signal Image Video Process. 2017;11:251–258. doi: 10.1007/s11760-016-0928-z. DOI

Motion Assessment for Accelerometric and Heart Rate Cycling Data Analysis

Sleep Apnea Detection with Polysomnography and Depth Sensors