This paper is devoted to a new method of using Microsoft (MS) Kinect sensors for non-contact monitoring of breathing and heart rate estimation to detect possible medical and neurological disorders. Video sequences of facial features and thorax movements are recorded by MS Kinect image, depth and infrared sensors to enable their time analysis in selected regions of interest. The proposed methodology includes the use of computational methods and functional transforms for data selection, as well as their denoising, spectral analysis and visualization, in order to determine specific biomedical features. The results that were obtained verify the correspondence between the evaluation of the breathing frequency that was obtained from the image and infrared data of the mouth area and from the thorax movement that was recorded by the depth sensor. Spectral analysis of the time evolution of the mouth area video frames was also used for heart rate estimation. Results estimated from the image and infrared data of the mouth area were compared with those obtained by contact measurements by Garmin sensors (www.garmin.com). The study proves that simple image and depth sensors can be used to efficiently record biomedical multidimensional data with sufficient accuracy to detect selected biomedical features using specific methods of computational intelligence. The achieved accuracy for non-contact detection of breathing rate was 0.26% and the accuracy of heart rate estimation was 1.47% for the infrared sensor. The following results show how video frames with depth data can be used to differentiate different kinds of breathing. The proposed method enables us to obtain and analyse data for diagnostic purposes in the home environment or during physical activities, enabling efficient human-machine interaction.

Czech Institute of Informatics Robotics and Cybernetics Czech Technical University 166 36 Prague 6 Czech Republic

Department of Computing and Control Engineering University of Chemistry and Technology 166 28 Prague 6 Czech Republic

Department of Neurology Faculty of Medicine in Hradec Kr and 225;lov and 233; Charles University 500 05 Hradec Kr and 225;lov and 233; Czech Republic

Zobrazit více v PubMed

Benetazzo F., Freddi A., Monteriu A., Longhi S. Respiratory rate detection algorithm based on RGB-D camera: Theoretical background and experimental results. Healthc. Technol. Lett. 2014;1:81–86. doi: 10.1049/htl.2014.0063. PubMed DOI PMC

Fekr A., Janidarmian M., Radecka K., Zilic Z. A Medical Cloud-Based Platform for Respiration Rate Measurement and Hierarchical Classification of Breath Disorders. Sensors. 2014;14:11204–11224. doi: 10.3390/s140611204. PubMed DOI PMC

Hong K., Hong S. Real-time stress assessment using thermal imaging. Vis. Comput. 2015;2015:1–9. doi: 10.1007/s00371-015-1164-1. DOI

Jing B., Li H. A Novel Thermal Measurement for Heart Rate. J. Comput. 2013;8:2163–2166. doi: 10.4304/jcp.8.9.2163-2166. DOI

Massagram W., Lubecke V., Host-Madsen A., Boric-Lubecke O. Assessment of Heart Rate Variability and Respiratory Sinus Arrhythmia via Doppler Radar. IEEE Trans. Microw. Theory Technol. 2009;57:2543–2543. doi: 10.1109/TMTT.2009.2029716. DOI

Monkaresi H., Calvo R.A., Yan H. A machine learning approach to improve contactless heart rate monitoring using a webcam. IEEE J. Biomed. Health Inform. 2014;18:1153–1160. doi: 10.1109/JBHI.2013.2291900. PubMed DOI

Shao D., Yang Y., Liu C., Tsow F., Yu H., Tao N. Noncontact monitoring breathing pattern, exhalation flow rate and pulse transit time. IEEE Trans. Bio-Med. Eng. 2014;61:2760–2767. doi: 10.1109/TBME.2014.2327024. PubMed DOI

Yang L., Ren Y., Hu H., Tian B. New fast fall detection method based on spatio-temporal context tracking of head by using depth images. Sensors. 2015;15:23004–23019. doi: 10.3390/s150923004. PubMed DOI PMC

Balakrishnan G., Durand F., Gutag J. Detecting pulse from head motions in video; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR; Portland, OR, USA. 20–25 June 2013; pp. 3430–3437.

Poh M., McDuff D.J., Picard R.W. Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Opt. Express. 2010;18:10762–10774. doi: 10.1364/OE.18.010762. PubMed DOI

Gao Z., Yu Y., Du S. Leveraging Two Kinect Sensors for Accurate Full-Body Motion Capture. Sensors. 2015;15:24297–24317. doi: 10.3390/s150924297. PubMed DOI PMC

Lachat E., Macher H., Landes T., Grussenmeyer P. Assessment and Calibration of a RGB-D Camera (Kinect v2 Sensor) Towards a Potential Use for Close-Range 3D Modeling. Remote Sens. 2015;7:13070–13097. doi: 10.3390/rs71013070. DOI

Procházka A., Vyšata O., Vališ M., Ťupa O., Schatz M., Mařík V. Bayesian Classification and Analysis of Gait Disorders Using Image and Depth Sensors of Microsoft Kinect. Digit. Signal Process. 2015;47:169–177. doi: 10.1016/j.dsp.2015.05.011. DOI

Procházka A., Vyšata O., Vališ M., Ťupa O., Schatz M., Mařík V. Use of Image and Depth Sensors of the Microsoft Kinect for the Detection of Gait Disorders. Neural Comput. Appl. 2015;26:1621–1629. doi: 10.1007/s00521-015-1827-x. DOI

Yang C., Cheung G., Stankovic V. Estimating heart rate via depth video motion tracking; Proceedings of the IEEE International Conference on Multimedia and Expo, ICME-2015; Torino, Italy. 29 June–3 July 2015; pp. 1–6.

Bae M., Park I. Content-based 3D model retrieval using a single depth image from a low-cost 3D camera. Vis. Comput. 2013;29:555–564. doi: 10.1007/s00371-013-0819-z. DOI

Ťupa O., Procházka A., Vyšata O., Schatz M., Mareš J., Vališ M., Mařík V. Motion tracking and gait feature estimation for recognising Parkinson’s disease using MS Kinect. BioMed. Eng. OnLine. 2015;14:1–20. doi: 10.1186/s12938-015-0092-7. PubMed DOI PMC

Yu M., Wu H., Liou J., Lee M., Hung Y. Multiparameter sleep monitoring using a depth camera; Proceedings of the BIOSTEC 2012; Vilamoura, Portugal. 1–4 February 2012; pp. 311–325.

Ma Z., Wu E. Real-time and robust hand tracking with a single depth camera. Vis. Comput. 2014;30:1133–1144. doi: 10.1007/s00371-013-0894-1. DOI

Alimohamed S., Prosser K., Weerasuriya C., Iles R., Cameron J., Lasenby J., Fogarty C. P134 Validating structured light plethysmography (SLP) as a non-invasive method of measuring lung function when compared to Spirometry. Thorax. 2011;66:A121. doi: 10.1136/thoraxjnl-2011-201054c.134. DOI

Brand D., Lau E., Cameron J., Wareham R., Usher-Smith J., Bridge P., Lasenby J., Iles R. Tidal Breathing Parameters Measurement by Structured Light Plethysmography (SLP) and Spirometry. Am. J. Resp. Crit. Care. 2010;B18:A2528–A2528.

Loblaw A., Nielsen J., Okoniewski M., Lakhani M.A. Remote respiratory sensing with an infrared camera using the Kinect infrared projector; Proceedings of the 2013 World Congress in Computer Science, Computer Engineering and Alied Computing, WORLDCOMP; Las Vegas, NV, USA. 25–28 July 2013; pp. 1–7.

Murthy R., Pavlidis I. Noncontact measurement of breathing function. IEEE Eng. Med. Biol. Mag. 2014;25:57–67. doi: 10.1109/MEMB.2006.1636352. PubMed DOI

Wang C.W., Hunter A., Gravill N., Matusiewicz S. Unconstrained video monitoring of breathing behavior and application to diagnosis of sleep apnea. IEEE Trans. Biomed. Eng. 2014;61:396–404. doi: 10.1109/TBME.2013.2280132. PubMed DOI

Falie D., Ichim M., David L. Respiratory motion visualization and the sleep apnea diagnosis with the time of flight (ToF) camera; Proceedings of the 1st WSEAS International Conference on VISUALIZATION, IMAGING and SIMULATION, VIS’08; Bucharest, Romania. 7–9 November 2008; pp. 179–184.

Gu C., Li C. Assessment of Human Respiration Patterns via Noncontact Sensing Using Doppler Multi-Radar System. Sensors. 2015;15:6383–6398. doi: 10.3390/s150306383. PubMed DOI PMC

Arlotto P., Grimaldi M., Naeck R., Ginoux J. An Ultrasonic Contactless Sensor for Breathing Monitoring. Sensors. 2014;14:15371–15386. doi: 10.3390/s140815371. PubMed DOI PMC

Hashizaki M., Nakajima H., Kume K. Monitoring of Weekly Sleep Pattern Variations at Home with a Contactless Biomotion Sensor. Sensors. 2014;14:18950–18964. doi: 10.3390/s150818950. PubMed DOI PMC

Pandiyan E.M., Selvan M.T., Hussian M.S., Velmathi D.G. Force Sensitive Resistance Based Heart Beat Monitoring for Health Care System. Int. J. Inform. Sci. Technol. 2014;4:11–16. doi: 10.5121/ijist.2014.4302. DOI

Jerhotová E., Švihlík J., Procházka A. Biomedical Image Volumes Denoising via the Wavelet Transform. In: Gargiulo G.D., McEwan A., editors. Applied Biomedical Engineering. INTECH; Rijeka, Croatia: 2011. pp. 435–458.

Procházka A., Vyšata O., Ťupa O., Mareš J., Vališ M. Discrimination of Axonal Neuropathy Using Sensitivity and Specificity Statistical Measures. Neural Comput. Appl. 2014;25:1349–1358. doi: 10.1007/s00521-014-1622-0. DOI

Cippitelli E., Gasparrini S., Gambi E., Spinsante S. A Human Activity Recognition System Using Skeleton Data from RGBD Sensors. Comput. Intell. Neurosci. 2016;2016:1–14. doi: 10.1155/2016/4351435. PubMed DOI PMC

Erden F., Velipasalar S., Alkar A., Cetin A. Sensors in Assisted Living. IEEE Signal Proc. Mag. 2016;33:36–44. doi: 10.1109/MSP.2015.2489978. DOI

Ye M., Yang C., Stankovic V., Stankovic L., Kerr A. A depth camera motion analysis framework for tele-rehabilitation: Motion capture and person-centric kinematics analysis. IEEE J. Sel. Top. Signal Process. 2016;2016:1–11. doi: 10.1109/JSTSP.2016.2559446. DOI

Yang C., Mao Y., Cheung G., Stankovic V., Chan K. Monitoring via Depth Video Recording & Analysis; Proceedings of the 5th IEEE International Workshop on Hot Topics in 3D, Hot3D; Chengdu, China. 14–18 July 2014; pp. 1–6.

Yang C., Mao Y., Cheung G., Stankovic V., Chan K. Graph-Based Depth Video Denoising and Event Detection for Sleep Monitoring; Proceedings of the IEEE International Workshop on Multimedia Signal Processing; Jakarta, Indonesia. 7–9 September 2014; pp. 1–6.

Bandyopadhyay S.K. Lip Contour Detection Techniques Based on Front View Of Face. J. Glob. Res. Comput. Sci. 2011;2:43–46.

Khan I., Abdullah H., Zainal M.S.B. Efficient Eyes and Mouth Detection Algorithm Using Combination of Viola Jones and Skin Color Pixel Detection. Int. J. Eng. Appl. Sci. 2013;3:51–60.

Khoshelham K., Elberink S.O. Accuracy and Resolution of Kinect Depth Data for Indoor Mapping Applications. Sensors. 2012;12:1437–1454. doi: 10.3390/s120201437. PubMed DOI PMC

Chen K., Lorenz D.A. Image Sequence Interpolation Using Optimal Control. J. Math. Imaging Vis. 2011;41:222–238. doi: 10.1007/s10851-011-0274-2. DOI

Zi L., Du J., Wang Q. Frame Interpolation Based on Visual Correspondence and Coherency Sensitive Hashing. Math. Probl. Eng. 2013;13:1–11. doi: 10.1155/2013/453278. DOI

Merget D., Eckl T., Schwörer M., Tiefenbacher P., Rigoll G. Capturing Facial Videos with Kinect 2.0: A Multithreaded Open Source Tool and Database; Proceedings of the IEEE Winter Conference on Applications of Computer Vision, WACV 2016; Lake Placid, NY, USA. 7–9 March 2016; pp. 1–5.

Breathing Analysis Using Thermal and Depth Imaging Camera Video Records