This paper is devoted to proving two goals, to show that various depth sensors can be used to record breathing rate with the same accuracy as contact sensors used in polysomnography (PSG), in addition to proving that breathing signals from depth sensors have the same sensitivity to breathing changes as in PSG records. The breathing signal from depth sensors can be used for classification of sleep [d=R2]apneaapnoa events with the same success rate as with PSG data. The recent development of computational technologies has led to a big leap in the usability of range imaging sensors. New depth sensors are smaller, have a higher sampling rate, with better resolution, and have bigger precision. They are widely used for computer vision in robotics, but they can be used as non-contact and non-invasive systems for monitoring breathing and its features. The breathing rate can be easily represented as the frequency of a recorded signal. All tested depth sensors (MS Kinect v2, RealSense SR300, R200, D415 and D435) are capable of recording depth data with enough precision in depth sensing and sampling frequency in time (20-35 frames per second (FPS)) to capture breathing rate. The spectral analysis shows a breathing rate between 0.2 Hz and 0.33 Hz, which corresponds to the breathing rate of an adult person during sleep. To test the quality of breathing signal processed by the proposed workflow, a neural network classifier (simple competitive NN) was trained on a set of 57 whole night polysomnographic records with a classification of sleep [d=R2]apneaapnoas by a sleep specialist. The resulting classifier can mark all [d=R2]apneaapnoa events with 100% accuracy when compared to the classification of a sleep specialist, which is useful to estimate the number of events per hour. [d=R2]When compared to the classification of polysomnographic breathing signal segments by a sleep specialistand, which is used for calculating length of the event, the classifier has an [d=R1] F 1 score of 92.2%Accuracy of 96.8% (sensitivity 89.1% and specificity 98.8%). The classifier also proves successful when tested on breathing signals from MS Kinect v2 and RealSense R200 with simulated sleep [d=R2]apneaapnoa events. The whole process can be fully automatic after implementation of automatic chest area segmentation of depth data.

Charles University University Hospital Hradec Králové Faculty of Medicine in Hradec Králové Department of Neurology 500 05 Hradec Králové Czech Republic

Czech Technical University Prague Czech Institute of Informatics Robotics and Cybernetics 160 00 Prague 6 Czech Republic

University of Chemistry and Technology Prague Department of Computing and Control Engineering 166 28 Prague 6 Czech Republic

Zobrazit více v PubMed

Schätz M., Centonze F., Kuchynka J., Tupa O., Vysata O., Geman O., Prochazka A. Statistical recognition of breathing by MS Kinect depth sensor; Proceedings of the 2015 International Workshop on Computational Intelligence for Multimedia Understanding (IWCIM); Prague, Czech Republic. 29–30 October 2015; pp. 1–4. DOI

Ťupa O., Procházka A., Vyšata O., Schatz M., Mareš J., Vališ M., Mařík V. Motion tracking and gait feature estimation for recognising Parkinson’s disease using MS Kinect. BioMed. Eng. OnLine. 2015;14:1–20. doi: 10.1186/s12938-015-0092-7. PubMed DOI PMC

Procházka A., Vyšata O., Vališ M., Ťupa O., Schatz M., Mařík V. Bayesian Classification and Analysis of Gait Disorders Using Image and Depth Sensors of Microsoft Kinect. Elsevier Digit. Signal Process. 2015;47:169–177. doi: 10.1016/j.dsp.2015.05.011. DOI

Procházka A., Vyšata O., Vališ M., Ťupa O., Schatz M., Mařík V. Use of Image and Depth Sensors of the Microsoft Kinect for the Detection of Gait Disorders. Springer Neural Comput. Appl. 2015;26:1621–1629. doi: 10.1007/s00521-015-1827-x. DOI

Lachat E., Macher H., Landes T., Grussenmeyer P. Assessment and Calibration of a RGB-D Camera (Kinect v2 Sensor) Towards a Potential Use for Close-Range 3D Modeling. Remote Sens. 2015;7:13070–13097. doi: 10.3390/rs71013070. DOI

Gao Z., Yu Y., Du S. Leveraging Two Kinect Sensors for Accurate Full-Body Motion Capture. Sensors. 2015;15:24297–24317. doi: 10.3390/s150924297. PubMed DOI PMC

Addison P.S., Smit P., Jacquel D., Borg U.R. Continuous respiratory rate monitoring during an acute hypoxic challenge using a depth sensing camera. J. Clin. Monit. Comput. 2019:1–9. doi: 10.1007/s10877-019-00417-6. PubMed DOI PMC

Carey G. How Intel’s RealSense Has Come of Age | Digital Trends. [(accessed on 2 March 2020)];2016 Available online: https://www.digitaltrends.com/computing/intel-realsense-coming-of-age/

Martinez M., Stiefelhagen R. Breathing rate monitoring during sleep from a depth camera under real-life conditions; Proceedings of the 2017 IEEE Winter Conference on Applications of Computer Vision, WACV 2017; Santa Rosa, CA, USA. 24–31 March 2017; pp. 1168–1176. DOI

Yang C., Cheung G., Stankovic V., Chan K., Ono N. Sleep Apnea Detection via Depth Video and Audio Feature Learning. IEEE Trans. Multimed. 2017;19:822–835. doi: 10.1109/TMM.2016.2626969. DOI

Procházka A., Charvátová H., Vyšata O., Kopal J., Chambers J. Breathing Analysis Using Thermal and Depth Imaging Camera Video Records. Sensors. 2017;17:1408. doi: 10.3390/s17061408. PubMed DOI PMC

Wang Y.K., Chen H.Y., Chen J.R. Unobtrusive Sleep Monitoring Using Movement Activity by Video Analysis. Electronics. 2019;8:812. doi: 10.3390/electronics8070812. DOI

Alimohamed S., Prosser K., Weerasuriya C., Iles R., Cameron J., Lasenby J., Fogarty C. P134 Validating structured light plethysmography (SLP) as a non-invasive method of measuring lung function when compared to Spirometry. Thorax. 2011;66:A121. doi: 10.1136/thoraxjnl-2011-201054c.134. DOI

Brand D., Lau E., Cameron J., Wareham R., Usher-Smith J., Bridge P., Lasenby J., Iles R. Tidal Breathing Parameters Measurement by Structured Light Plethysmography (SLP) and Spirometry. Am. J. Respir. Crit. Care Med. 2010;B18:A2528.

Wang C.W., Hunter A., Gravill N., Matusiewicz S. Unconstrained video monitoring of breathing behavior and application to diagnosis of sleep apnea. IEEE Trans. Biomed. Eng. 2014;61:396–404. doi: 10.1109/TBME.2013.2280132. PubMed DOI

Murthy R., Pavlidis I. Noncontact measurement of breathing function. IEEE Eng. Med. Biol. Mag. 2014;25:57–67. doi: 10.1109/MEMB.2006.1636352. PubMed DOI

Gu C., Li C. Assessment of Human Respiration Patterns via Noncontact Sensing Using Doppler Multi-Radar System. Sensors. 2015;15:6383–6398. doi: 10.3390/s150306383. PubMed DOI PMC

Arlotto P., Grimaldi M., Naeck R., Ginoux J. An Ultrasonic Contactless Sensor for Breathing Monitoring. Sensors. 2014;14:15371–15386. doi: 10.3390/s140815371. PubMed DOI PMC

Hashizaki M., Nakajima H., Kume K. Monitoring of Weekly Sleep Pattern Variations at Home with a Contactless Biomotion Sensor. Sensors. 2014;15:18950–18964. doi: 10.3390/s150818950. PubMed DOI PMC

Pandiyan E., Selvan M., Hussian M., Velmathi D. Force Sensitive Resistance Based Heart Beat Monitoring For Health Care System. Int. J. Inf. Sci. Technol. 2014;4:11–16. doi: 10.5121/ijist.2014.4302. DOI

Nam Y., Kim Y., Lee J. Sleep Monitoring Based on a Tri-Axial Accelerometer and a Pressure Sensor. Sensors. 2016;16:750. doi: 10.3390/s16050750. PubMed DOI PMC

Cippitelli E., Gasparrini S., Gambi E., Spinsante S. A Human Activity Recognition System Using Skeleton Data from RGBD Sensors. Comput. Intell. Neurosci. 2016;2016:1–14. doi: 10.1155/2016/4351435. ID 4351435. PubMed DOI PMC

Erden F., Velipasalar S., Alkar A., Cetin A. Sensors in Assisted Living. IEEE Signal Process. Mag. 2016;33:36–44. doi: 10.1109/MSP.2015.2489978. PubMed DOI

Ye M., Yang C., Stankovic V., Stankovic L., Kerr A. A depth camera motion analysis framework for tele-rehabilitation: Motion capture and person-centric kinematics analysis. IEEE J. Sel. Top. Signal Process. 2016;2016:1–11. doi: 10.1109/JSTSP.2016.2559446. DOI

Rolls E., Deco G. Computational Neuroscience of Vision. Oxford University Press; Oxford, UK: 2001. DOI

Salatas J. Implementation of Competitive Learning Networks for WEKA - ICT Research Blog. [(accessed on 1 March 2020)];2011 Available online: https://jsalatas.ictpro.gr/implementation-of-competitive-learning-networks-for-weka/

Winograd S. On computing the discrete Fourier transform. Math. Comput. 1978;32:175. doi: 10.1090/S0025-5718-1978-0468306-4. PubMed DOI PMC

Bradley T.D., McNicholas W.T., Rutherford R., Popkin J., Zamel N., Phillipson E.A. Clinical and physiologic heterogeneity of the central sleep apnea syndrome. Am. Rev. Respir. Dis. 1981;305:325–330. PubMed

Guilleminault C., Eldridge F.L., Dement W.C. Insomnia with sleep apnea: A new syndrome. Science. 1973;181:856–858. doi: 10.1126/science.181.4102.856. PubMed DOI

Guilleminault C., Quera-Salva M.A., Nino-Murcia G., Partinen M. Central sleep apnea and partial obstruction of the upper airway. Ann. Neurol. 1987;21:465–469. doi: 10.1002/ana.410210509. PubMed DOI

Issa F.G., Sullivan C.E. Reversal of central sleep apnea using nasal CPAP. Chest. 1986;90:165–171. doi: 10.1378/chest.90.2.165. PubMed DOI

Cherniack N.S. Respiratory dysrhythmias during sleep. N. Engl. J. Med. 1981;305:325–330. PubMed

Thorpy M.J., Rochester M. International Classification of Sleep Disorders: Diagnostic and Coding Manual, Revised. American Academy of Sleep Medicine; Rochester, MN, USA: 1997. pp. 182–185.

Schätz M., Kuchyňka J., Vyšata O., Procházka A. Pilot Study of Sleep Apnea Detection with Wavelet Transform; Technical Computing Prague; 2017. [(accessed on 1 March 2020)]; Available online: https://pdfs.semanticscholar.org/3a1e/893519193746565bc2559d5181c29c95e422.pdf.

Deboer S.L. Emergency Newborn Care. Trafford; Bloomington, IN, USA: 2006. p. 170.

Lindh W., Pooler M., Tamparo C., Dahl B. Delmar’s Comprehensive Medical Assisting: Administrative and Clinical Competencies. Cengage Learning; Belmont, CA, USA: 2009. p. 1552.

Lee Y.S., Pathirana P.N., Member S., Steinfort C.L. Respiration Rate and Breathing Patterns from Doppler Radar Measurements; Proceedings of the 2014 IEEE Conference on Biomedical Engineering and Sciences (IECBES); Kuala Lumpur, Malaysia. 8–10 December 2014; pp. 8–10.

Barrett K.E., Barman S.M., Boitano S., Brooks H. Ganong’s Review of Medical Physiology. 24th ed. McGraw-Hill Education; Berkshire, UK: 2012.

Rodríguez-Molinero A., Narvaiza L., Ruiz J., Gálvez-Barrón C. Normal respiratory rate and peripheral blood oxygen saturation in the elderly population. J. Am. Geriatr. Soc. 2013;61:2238–2240. doi: 10.1111/jgs.12580. PubMed DOI

Wasenmüller O., Stricker D. Comparison of Kinect V1 and V2 Depth Images in Terms of Accuracy and Precision. Springer; Berlin, Germany: 2017. pp. 34–45. DOI

Keselman L., Woodfill J.I., Grunnet-Jepsen A., Bhowmik A. Intel RealSense Stereoscopic Depth Cameras. arXiv. 20171705.05548

Intel . Intel ® RealSense™ Camera R200 Embedded Infrared Assisted Stereovision 3D Imaging System with Color Camera Product Datasheet R200. 2016. [(accessed on 1 March 2020)]. Technical Report. Available online: https://www.mouser.com/pdfdocs/intel_realsense_camera_r200.pdf.