Deep learning has recently been utilized with great success in a large number of diverse application domains, such as visual and face recognition, natural language processing, speech recognition, and handwriting identification. Convolutional neural networks, that belong to the deep learning models, are a subtype of artificial neural networks, which are inspired by the complex structure of the human brain and are often used for image classification tasks. One of the biggest challenges in all deep neural networks is the overfitting issue, which happens when the model performs well on the training data, but fails to make accurate predictions for the new data that is fed into the model. Several regularization methods have been introduced to prevent the overfitting problem. In the research presented in this manuscript, the overfitting challenge was tackled by selecting a proper value for the regularization parameter dropout by utilizing a swarm intelligence approach. Notwithstanding that the swarm algorithms have already been successfully applied to this domain, according to the available literature survey, their potential is still not fully investigated. Finding the optimal value of dropout is a challenging and time-consuming task if it is performed manually. Therefore, this research proposes an automated framework based on the hybridized sine cosine algorithm for tackling this major deep learning issue. The first experiment was conducted over four benchmark datasets: MNIST, CIFAR10, Semeion, and UPS, while the second experiment was performed on the brain tumor magnetic resonance imaging classification task. The obtained experimental results are compared to those generated by several similar approaches. The overall experimental results indicate that the proposed method outperforms other state-of-the-art methods included in the comparative analysis in terms of classification error and accuracy.

Artificial Intelligence Engineering Department Research Center for AI and IoT AI and Robotics Institute Near East University Mersin 10 Turkey

Department of Applied Cybernetics Faculty of Science University of Hradec Králové 50003 Hradec Králové Czech Republic

Department of Mathematics Faculty of Science University of Hradec Králové 50003 Hradec Králové Czech Republic

Singidunum University Danijelova 32 11000 Belgrade Serbia

Zobrazit více v PubMed

McCarthy J. From here to human-level AI. Artif. Intell. 2007;171(18):1174–1182. doi: 10.1016/j.artint.2007.10.009. DOI

Arel I, Rose DC, Karnowski TP. Deep machine learning-a new frontier in artificial intelligence research [research frontier] IEEE Comput. Intell. Mag. 2010;5(4):13–18. doi: 10.1109/MCI.2010.938364. DOI

Hongtao L, Qinchuan Z. Applications of deep convolutional neural network in computer vision. J. Data Acquis. Process. 2016;31(1):1–17.

Xiao, T., Xu, Y., Yang, K., Zhang, J., Peng, Y. & Zhang, Z. The application of two-level attention models in deep convolutional neural network for fine-grained image classification, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, 842–850.

Zhang Y, Zhao D, Sun J, Zou G, Li W. Adaptive convolutional neural network and its application in face recognition. Neural Process. Lett. 2016;43(2):389–399. doi: 10.1007/s11063-015-9420-y. DOI

LeCun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proc. IEEE. 1998;86(11):2278–2324. doi: 10.1109/5.726791. DOI

de Rosa G, Papa J, Yang X-S. Handling dropout probability estimation in convolution neural networks using metaheuristics. Soft Comput. 2018 doi: 10.1007/s00500-017-2678-4. DOI

Bacanin N, Bezdan T, Tuba E, Strumberger I, Tuba M. Monarch butterfly optimization based convolutional neural network design. Mathematics. 2020;8(6):936. doi: 10.3390/math8060936. DOI

Sammut, C. & Webb, G. I. (Eds.), Bias-Variance Trade-offs, Springer US, Boston, MA, 110, (2010). 10.1007/978-0-387-30164-8_76.

Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R. Dropout: A simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 2014;15(1):1929–1958.

Bacanin, N., Tuba, E., Bezdan, T., Strumberger, I., Jovanovic, R. & Tuba M. Dropout probability estimation in convolutional neural networks by the enhanced bat algorithm, in 2020 International Joint Conference on Neural Networks (IJCNN), IEEE, 1–7, (2020).

Mirjalili S. Sca: A sine cosine algorithm for solving optimization problems. Knowl.-Based Syst. 2016;96:120–133. doi: 10.1016/j.knosys.2015.12.022. DOI

Gabis AB, Meraihi Y, Mirjalili S, Ramdane-Cherif A. A comprehensive survey of sine cosine algorithm: Variants and applications. Artif. Intell. Rev. 2021;1–72:5469–5540. doi: 10.1007/s10462-021-10026-y. PubMed DOI PMC

Li Y, Zhao Y, Liu J. Dynamic sine cosine algorithm for large-scale global optimization problems. Expert Syst. Appl. 2021;177:114950. doi: 10.1016/j.eswa.2021.114950. DOI

Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 2012;25:1097–1105.

Lawrence S, Giles CL, Tsoi AC, Back AD. Face recognition: A convolutional neural-network approach. IEEE Trans. Neural Netw. 1997;8(1):98–113. doi: 10.1109/72.554195. PubMed DOI

Ranjan, R., Sankaranarayanan, S., Castillo, C. D. & Chellappa, R. An all-in-one convolutional neural network for face analysis, in 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), IEEE, 17–24, (2017).

Matsugu M, Mori K, Mitari Y, Kaneda Y. Subject independent facial expression recognition with robust face detection using a convolutional neural network. Neural Netw. 2003;16(5–6):555–559. doi: 10.1016/S0893-6080(03)00115-1. PubMed DOI

Ramaiah, N. P., Ijjina, E. P. Mohan, C. K. Illumination invariant face recognition using convolutional neural networks, in 2015 IEEE International Conference on Signal Processing, Informatics, Communication and Energy Systems (SPICES), IEEE, 1–4, (2015).

Simard, P.Y., Steinkraus, D. & Platt, J. C. et al., Best practices for convolutional neural networks applied to visual document analysis., in: Icdar, Vol. 3, Citeseer, (2003).

Afzal, M. Z. et al. Deepdocclassifier: Document classification with deep convolutional neural network, in 2015 13th International Conference on Document Analysis and Recognition (ICDAR). IEEE, 1111–1115 (2015).

Špetlík, R., Franc, V. & Matas, J. Visual heart rate estimation with convolutional neural network, in Proceedings of the British Machine Vision Conference, Newcastle, UK, 3–6, (2018).

Li, Q. et al. Medical image classification with convolutional neural network, in 2014, 13th International Conference on Control Automation Robotics & Vision (ICARCV). IEEE, 844–848 (2014).

Ting FF, Tan YJ, Sim KS. Convolutional neural network improvement for breast cancer classification. Expert Syst. Appl. 2019;120:103–115. doi: 10.1016/j.eswa.2018.11.008. DOI

Liu, Y., Racah, E., Correa, J., Khosrowshahi, A., Lavers, D., Kunkel, K., Wehner, M., Collins, W. et al., Application of deep convolutional neural networks for detecting extreme weather in climate datasets, arXiv preprint arXiv:1605.01156, (2016).

Chattopadhyay A, Hassanzadeh P, Pasha S. Predicting clustered weather patterns: A test case for applications of convolutional neural networks to spatio-temporal climate data. Sci. Rep. 2020;10(1):1–13. doi: 10.1038/s41598-020-57897-9. PubMed DOI PMC

Gavrilov AD, Jordache A, Vasdani M, Deng J. Preventing model overfitting and underfitting in convolutional neural networks. Int. J. Softw. Sci. Comput. Intell. (IJSSCI) 2018;10(4):19–28. doi: 10.4018/IJSSCI.2018100102. DOI

Ng, A. Y. Feature selection, l 1 versus l 2 regularization, and rotational invariance, in Proceedings of the Twenty-First International Conference on Machine learning, 78, (2004).

Yang X-S. Recent Advances in Swarm Intelligence and Evolutionary Computation. NewYork: Springer; 2015.

Zivkovic, M. et al. Wireless sensor networks life time optimization based on the improved firefly algorithm, in 2020, International Wireless Communications and Mobile Computing (IWCMC). IEEE, 1176–1181 (2020).

Zivkovic, M. et al. Enhanced grey wolf algorithm for energy efficient wireless sensor networks, in f2020 Zooming Innovation in Consumer Technologies Conference (ZINC). IEEE, 87–92 (2020).

Bacanin, N., Tuba, E., Zivkovic, M., Strumberger, I. & Tuba, M. Whale optimization algorithm with exploratory move for wireless sensor networks localization, in International Conference on Hybrid Intelligent Systems, Springer, 328–338, (2019).

Zivkovic, M., Zivkovic, T., Venkatachalam, K. & Bacanin, N. Enhanced dragonfly algorithm adapted for wireless sensor network lifetime optimization, in Data Intelligence and Cognitive Informatics, Springer, 803–817, (2021).

Bacanin, N. et al. Task scheduling in cloud computing environment by grey wolf optimizer, in 2019, 27th Telecommunications Forum (TELFOR). IEEE, 1–4 (2019).

Strumberger I, Bacanin N, Tuba M, Tuba E. Resource scheduling in cloud computing based on a hybridized whale optimization algorithm. Appl. Sci. 2019;9(22):4893. doi: 10.3390/app9224893. DOI

Brajevic, I., Tuba, M. & Bacanin, N. Multilevel image thresholding selection based on the cuckoo search algorithm, in Proceedings of the 5th International Conference on Visualization, Imaging and Simulation (VIS’12), Sliema, Malta, 217–222, (2012).

Tuba, E., Strumberger, I., Zivkovic, D., Bacanin, N. & Tuba, M. Mobile robot path planning by improved brain storm optimization algorithm, in 2018, IEEE Congress on Evolutionary Computation (CEC). IEEE, 1–8 (2018).

Bezdan, T., Zivkovic, M., Tuba, E., Strumberger, I., Bacanin, N. & Tuba, M. Glioma brain tumor grade classification from MRI using convolutional neural networks designed by modified fa, in International Conference on Intelligent and Fuzzy Systems, Springer, 955–963, (2020).

Bacanin N, Bezdan T, Venkatachalam K, Al-Turjman F. Optimized convolutional neural network by firefly algorithm for magnetic resonance image classification of glioma brain tumor grade. J. Real-Time Image Process. 2021;18(4):1–14. doi: 10.1007/s11554-021-01106-x. DOI

Bezdan, T., Cvetnic, D., Gajic, L., Zivkovic, M., Strumberger, I. & Bacanin, N. Feature selection by firefly algorithm with improved initialization strategy, in 7th Conference on the Engineering of Computer Based Systems, 1–8, (2021).

Zivkovic M, Bacanin N, Venkatachalam K, Nayyar A, Djordjevic A, Strumberger I, Al-Turjman F. Covid-19 cases prediction by using hybrid machine learning and beetle antennae search approach. Sustain. Cities Soc. 2021;66:102669. doi: 10.1016/j.scs.2020.102669. PubMed DOI PMC

Zivkovic, M., Venkatachalam, K., Bacanin, N., Djordjevic, A., Antonijevic, M., Strumberger, I. & Rashid, T. A. Hybrid genetic algorithm and machine learning method for covid-19 cases prediction, in Proceedings of International Conference on Sustainable Expert Systems: ICSES 2020, Vol. 176, Springer Nature, 169, (2021).

Milosevic, S., Bezdan, T., Zivkovic, M., Bacanin, N., Strumberger, I. & Tuba, M. Feed-forward neural network training by hybrid bat algorithm, in Modelling and Development of Intelligent Systems: 7th International Conference, MDIS 2020, Sibiu, Romania, October 22–24, 2020, Revised Selected Papers 7, Springer International Publishing, 52–66, (2021).

Gajic, L., Cvetnic, D., Zivkovic, M., Bezdan, M., Bacanin, N. & Milosevic, S. Multi-layer perceptron training using hybridized bat algorithm, in Computational Vision and Bio-Inspired Computing, Springer, 689–705, (2021).

Tuba M, Bacanin N. Artificial bee colony algorithm hybridized with firefly algorithm for cardinality constrained mean-variance portfolio selection problem. Appl. Math. Inf. Sci. 2014;8(6):2831. doi: 10.12785/amis/080619. DOI

Bacanin, N. & Tuba, M. Firefly algorithm for cardinality constrained mean-variance portfolio optimization problem with entropy diversity constraint, Sci. World J, special issue Computational Intelligence and Metaheuristic Algorithms with Applications, 2014 (Article ID 721521) (2014) 16. 10.1155/2014/721521. PubMed PMC

Tizhoosh, H.R. Opposition-based learning: A new scheme for machine intelligence, in: International Conference on Computational Intelligence for Modelling, Control and Automation and International Conference on Intelligent Agents, Web Technologies and Internet Commerce (CIMCA-IAWTIC’06), 1, 695–701, (2005).

Yang X-S. Firefly Algorithms for Multimodal Optimization. In: Watanabe O, Zeugmann T, editors. Stochastic Algorithms: Foundations and Applications. Berlin, Heidelberg: Springer; 2009. pp. 169–178.

Yang X-S, Xingshi H. Firefly algorithm: Recent advances and applications. Int. J. Swarm Intell. 2013;1(1):36–50. doi: 10.1504/IJSI.2013.055801. DOI

Price, K., Awad, N., Ali, N. Suganthan, P. Problem definitions and evaluation criteria for the 100-digit challenge special session and competition on single objective numerical optimization, in Technical Report, Nanyang Technological University, (2018).

Muthusamy H, Ravindran S, Yaacob S, Polat K. An improved elephant herding optimization using sine-cosine mechanism and opposition based learning for global optimization problems. Expert Syst. Appl. 2021;172:114607. doi: 10.1016/j.eswa.2021.114607. DOI

Friedman M. The use of ranks to avoid the assumption of normality implicit in the analysis of variance. J. Am. Stat. Assoc. 1937;32(200):675–701. doi: 10.1080/01621459.1937.10503522. DOI

Friedman M. A comparison of alternative tests of significance for the problem of m rankings. Ann. Math. Stat. 1940;11(1):86–92. doi: 10.1214/aoms/1177731944. DOI

Sheskin DJ. Handbook of Parametric and Nonparametric Statistical Procedures. NewYork: Chapman and Hall/CRC; 2020.

Iman RL, Davenport JM. Approximations of the critical region of the Fbietkan statistic. Commun. Stat.-Theor. Methods. 1980;9(6):571–595. doi: 10.1080/03610928008827904. DOI

Yang X-S, Gandomi A. H Bat algorithm: A novel approach for global engineering optimization. Eng. Comput. 2012;29(5):464–483. doi: 10.1108/02644401211235834. DOI

Gandomi AH, Yang X-S, Alavi AH. Cuckoo search algorithm: A metaheuristic approach to solve structural optimization problems. Eng. Comput. 2013;29(1):17–35. doi: 10.1007/s00366-011-0241-y. DOI

Kennedy, J. & Eberhart, R. Particle swarm optimization, in Proceedings of ICNN’95-international conference on neural networks, Vol. 4, IEEE, 1942–1948, (1995).

Wang G-G, Deb S, Gao X-Z, Coelho LDS. A new metaheuristic optimisation algorithm motivated by elephant herding behaviour. Int. J. Bio-Inspir. Comput. 2016;8(6):394–409. doi: 10.1504/IJBIC.2016.081335. DOI

Mirjalili S, Lewis A. The whale optimization algorithm. Adv. Eng. Softw. 2016;95:51–67. doi: 10.1016/j.advengsoft.2016.01.008. DOI

Mirjalili S, Gandomi AH, Mirjalili SZ, Saremi S, Faris H, Mirjalili SM. Salp swarm algorithm: A bio-inspired optimizer for engineering design problems. Adv. Eng. Softw. 2017;114:163–191. doi: 10.1016/j.advengsoft.2017.07.002. DOI

Mirjalili SZ, Mirjalili S, Saremi S, Faris H, Aljarah I. Grasshopper optimization algorithm for multi-objective optimization problems. Appl. Intell. 2018;48(4):805–820. doi: 10.1007/s10489-017-1019-8. DOI

Simon D. Biogeography-based optimization. IEEE Trans. Evolu. Comput. 2008;12(6):702–713. doi: 10.1109/TEVC.2008.919004. DOI

Anaraki AK, Ayati M, Kazemi F. Magnetic resonance imaging-based brain tumor grades classification and grading via convolutional neural networks and genetic algorithms. Biocybern. Biomed. Eng. 2019;39(1):63–74. doi: 10.1016/j.bbe.2018.10.004. DOI

Cheng J, Huang W, Cao S, Yang R, Yang W, Yun Z, Wang Z, Feng Q. Enhanced performance of brain tumor classification via tumor region augmentation and partition. PloS One. 2015;10(10):e0140381. doi: 10.1371/journal.pone.0140381. PubMed DOI PMC

Basha J, Bacanin N, Vukobrat N, Zivkovic M, Venkatachalam K, Hubálovskỳ S, Trojovskỳ P. Chaotic harris hawks optimization with quasi-reflection-based learning: An application to enhance CNN design. Sensors. 2021;21(19):6654. doi: 10.3390/s21196654. PubMed DOI PMC

Kalbkhani H, Shayesteh MG, Zali-Vargahan B. Robust algorithm for brain magnetic resonance image (MRI) classification based on Garch variances series. Biomed. Sig. Process. Control. 2013;8(6):909–919. doi: 10.1016/j.bspc.2013.09.001. DOI

Zacharaki EI, Wang S, Chawla S, Yoo DS, Wolf R, Melhem ER, Davatzikos C. Classification of brain tumor type and grade using mri texture and shape in a machine learning scheme. Magn. Reson. Med.: An Off. J. Int. Soc. Magn. Reson. Med. 2009;62(6):1609–1618. doi: 10.1002/mrm.22147. PubMed DOI PMC

Lecun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proc. IEEE. 1998;86(11):2278–2324. doi: 10.1109/5.726791. DOI

Simonyan, K. & Zisserman, A. Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv:1409.1556 (2014).

Huang, G., Liu, Z., Van Der Maaten, L. & Weinberger, K.Q. Densely connected convolutional networks, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4700–4708, (2017).