BACKGROUND: Type-1 diabetes is a condition caused by the lack of insulin hormone, which leads to an excessive increase in blood glucose level. The glucose kinetics process is difficult to control due to its complex and nonlinear nature and with state variables that are difficult to measure. METHODS: This paper proposes a method for automatically calculating the basal and bolus insulin doses for patients with type-1 diabetes using reinforcement learning with feedforward controller. The algorithm is designed to keep the blood glucose stable and directly compensate for the external events such as food intake. Its performance was assessed using simulation on a blood glucose model. The usage of the Kalman filter with the controller was demonstrated to estimate unmeasurable state variables. RESULTS: Comparison simulations between the proposed controller with the optimal reinforcement learning and the proportional-integral-derivative controller show that the proposed methodology has the best performance in regulating the fluctuation of the blood glucose. The proposed controller also improved the blood glucose responses and prevented hypoglycemia condition. Simulation of the control system in different uncertain conditions provided insights on how the inaccuracies of carbohydrate counting and meal-time reporting affect the performance of the control system. CONCLUSION: The proposed controller is an effective tool for reducing postmeal blood glucose rise and for countering the effects of external known events such as meal intake and maintaining blood glucose at a healthy level under uncertainties.

Czech Technical University Prague Czech Republic

The University of Melbourne Australia

UiT The Arctic University of Norway Tromsø Norway

Zobrazit více v PubMed

Wang Q., Molenaar P., Harsh S., et al. Personalized state-space modeling of glucose dynamics for type 1 diabetes using continuously monitored glucose, insulin dose, and meal intake. Journal of Diabetes Science and Technology. 2014;8(2):331–345. doi: 10.1177/1932296814524080. PubMed DOI PMC

Marchetti G., Barolo M., Jovanovic L., Zisser H., Seborg D. E. An improved PID switching control strategy for type 1 diabetes. IEEE Transactions on Biomedical Engineering. 2008;55(3):857–865. doi: 10.1109/tbme.2008.915665. PubMed DOI

Soylu S., Danisman K., Sacu I. E., Alci M. Closed-loop control of blood glucose level in type-1 diabetics: a simulation study. Proceedings of International Conference on Electrical and Electronics Engineering (ELECO); November 2013; Bursa, Turkey. pp. 371–375.

Boiroux D., Duun-Henriksen A. K., Schmidt S., et al. Overnight glucose control in people with type 1 diabetes. Biomedical Signal Processing and Control. 2018;39:503–512. doi: 10.1016/j.bspc.2017.08.005. DOI

Lee H., Bequette B. W. A closed-loop artificial pancreas based on model predictive control: human-friendly identification and automatic meal disturbance rejection. Biomedical Signal Processing and Control. 2009;4(4):347–354. doi: 10.1016/j.bspc.2009.03.002. DOI

Bothe M. K., Dickens L., Reichel K., et al. The use of reinforcement learning algorithms to meet the challenges of an artificial pancreas. Expert Review of Medical Devices. 2014;10(5):661–673. doi: 10.1586/17434440.2013.827515. PubMed DOI

De Paula M., Ávila L. O., Martínez E. C. Controlling blood glucose variability under uncertainty using reinforcement learning and Gaussian processes. Applied Soft Computing. 2015;35:310–332. doi: 10.1016/j.asoc.2015.06.041. DOI

Watkins C. J. C. H., Dayan P. Reinforcement Learning. Vol. 292. Boston, MA, USA: Springer US; 1992. Technical note: Q-learning; pp. 55–68. DOI

Pineau J., Bellemare M. G., Rush A. J., Ghizaru A., Murphy S. A. Constructing evidence-based treatment strategies using methods from computer science. Drug and Alcohol Dependence. 2007;88(S2):S52–S60. doi: 10.1016/j.drugalcdep.2007.01.005. PubMed DOI PMC

Lunze K., Singh T., Walter M., Brendel M. D., Leonhardt S. Blood glucose control algorithms for type 1 diabetic patients: a methodological review. Biomedical Signal Processing and Control. 2013;8(2):107–119. doi: 10.1016/j.bspc.2012.09.003. DOI

Bhattacharyyta S. P. Disturbance rejection in linear systems. International Journal of Systems Science. 1974;5(7):633–637. doi: 10.1080/00207727408920129. DOI

Zhong H., Pao L., de Callafon R. Feedforward control for disturbance rejection: model matching and other methods. Proceedings of 24th Chinese Control and Decision Conference (CCDC); May 2012; Taiyuan, China. pp. 3528–3533.

Lewis F. Optimal Estimation. Hoboken, NJ, USA: John Wiley & Sons, Inc.; 1986.

Franklin G. F., Powell J. D., Workman M. L. Digital Control of Dynamic Systems. 2nd. Boston, MA, USA: Addison-Wesley; 1990.

Vrabie D., Vamvoudakis K. G., Lewis F. L. Optimal Adaptive Control and Differential Games by Reinforcement Learning Principles. 1st. Vol. 81. London, UK: Institution of Engineering and Technology; 2012.

Ngo P. D., Wei S., Holubova A., Muzik J., Godtliebsen F. Reinforcement-learning optimal control for type-1 diabetes. Proceedings of 2018 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI); March 2018; Las Vegas, NV, USA. pp. 333–336.

Sutton R., Barto A. Reinforcement Learning: An Introduction. 1st. Cambridge, MA, USA: MIT Press; 1998.

MathWorks. MATLAB Optimization Toolbox: User’s Guide (r2018a) Natick, MA, USA: MathWorks; 2018.

Brazeau A. S., Mircescu H., Desjardins K., et al. Carbohydrate counting accuracy and blood glucose variability in adults with type 1 diabetes. Diabetes Research and Clinical Practice. 2013;99(1):19–23. doi: 10.1016/j.diabres.2012.10.024. PubMed DOI

Bergman R. N., Ider Y. Z., Bowden C. R., Cobelli C. Quantitative estimation of insulin sensitivity. American Journal of Physiology-Endocrinology and Metabolism. 1979;236(6):p. E667. doi: 10.1152/ajpendo.1979.236.6.e667. PubMed DOI

Hovorka R., Canonico V., Chassin L. J., et al. Nonlinear model predictive control of glucose concentration in subjects with type 1 diabetes. Physiological Measurement. 2004;25(4):905–920. doi: 10.1088/0967-3334/25/4/010. PubMed DOI

Wilinska M. E., Chassin L. J., Schaller H. C., Schaupp L., Pieber T. R., Hovorka R. Insulin kinetics in type-1 diabetes: continuous and bolus delivery of rapid acting insulin. IEEE Transactions on Biomedical Engineering. 2005;52(1):3–12. doi: 10.1109/tbme.2004.839639. PubMed DOI

Mösching A. Reinforcement Learning Methods for Glucose Regulation in Type 1 Diabetes. Lausanne, Switzerland: Ecole Polytechnique Federale de Lausanne; 2016.