Colonoscopy is a reliable diagnostic method to detect colorectal polyps early on and prevent colorectal cancer. The current examination techniques face a significant challenge of high missed rates, resulting in numerous undetected polyps and irregularities. Automated and real-time segmentation methods can help endoscopists to segment the shape and location of polyps from colonoscopy images in order to facilitate clinician's timely diagnosis and interventions. Different parameters like shapes, small sizes of polyps, and their close resemblance to surrounding tissues make this task challenging. Furthermore, high-definition image quality and reliance on the operator make real-time and accurate endoscopic image segmentation more challenging. Deep learning models utilized for segmenting polyps, designed to capture diverse patterns, are becoming progressively complex. This complexity poses challenges for real-time medical operations. In clinical settings, utilizing automated methods requires the development of accurate, lightweight models with minimal latency, ensuring seamless integration with endoscopic hardware devices. To address these challenges, in this study a novel lightweight and more generalized Enhanced Nanonet model, an improved version of Nanonet using NanonetB for real-time and precise colonoscopy image segmentation, is proposed. The proposed model enhances the performance of Nanonet using Nanonet B on the overall prediction scheme by applying data augmentation, Conditional Random Field (CRF), and Test-Time Augmentation (TTA). Six publicly available datasets are utilized to perform thorough evaluations, assess generalizability, and validate the improvements: Kvasir-SEG, Endotect Challenge 2020, Kvasir-instrument, CVC-ClinicDB, CVC-ColonDB, and CVC-300. Through extensive experimentation, using the Kvasir-SEG dataset, our model achieves a mIoU score of 0.8188 and a Dice coefficient of 0.8060 with only 132,049 parameters and employing minimal computational resources. A thorough cross-dataset evaluation was performed to assess the generalization capability of the proposed Enhanced Nanonet model across various publicly available polyp datasets for potential real-world applications. The result of this study shows that using CRF (Conditional Random Fields) and TTA (Test-Time Augmentation) enhances performance within the same dataset and also across diverse datasets with a model size of just 132,049 parameters. Also, the proposed method indicates improved results in detecting smaller and sessile polyps (flats) that are significant contributors to the high miss rates.

Department of Computer and Software Engineering College of Electrical and Mechanical Engineering Islamabad Pakistan

Department of Computer Science Sir Syed Institute of Technology Islamabad Pakistan

Department of Information and Communication Technology in Medicine Faculty of Biomedical Engineering Czech Technical University Prague Prague Czechia

Department of Mechanical and Electrical Systems Engineering Faculty of Engineering Kyoto University of Advanced Science Kyoto Japan

Department of Mechanical Engineering College of Electrical and Mechanical Engineering Islamabad Pakistan

Department of Physical Education Physical Education College of Zhengzhou University Zhengzhou China

Institute for Brain Research and Rehabilitation South China Normal University Guangzhou China

Laboratory of Human Factors and Automation in Aviation Department of Air Transport Faculty of Transportation Sciences Czech Technical University Prague Prague Czechia

School of Interdisciplinary Engineering and Sciences Islamabad Pakistan

School of Mechanical and Manufacturing Engineering Islamabad Pakistan

See more in PubMed

Aarons C. B., Shanmugan S., Bleier J. I. S. (2014). Management of malignant colon polyps: current status and controversies. World J. Gastroenterology WJG 20 (43), 16178. 10.3748/WJG.V20.I43.16178 PubMed DOI PMC

Abadi M., Barham P., Chen J., Chen Z., Davis A., Dean J., et al. (2016). TensorFlow: A system for large-scale machine learning. Available at: https://arxiv.org/abs/1605.08695v2 (Accessed December 18, 2023).

Alam F. I., Zhou J., Liew A. W. C., Jia X., Chanussot J., Gao Y. (2019). Conditional Random Field and Deep Feature Learning for hyperspectral Image Classification. IEEE Trans. Geoscience Remote Sens. 57 (3), 1612–1628. 10.1109/TGRS.2018.2867679 DOI

Alom M. Z., Hasan M., Yakopcic C., Taha T. M., Arasi V. K. (2018). Recurrent Residual Convolutional Neural Network based on U-Net (R2U-Net) for Medical Image Segmentation. Available at: https://arxiv.org/abs/1802.06955v5 (Accessed December 18, 2023).

Ameling S., Wirth S., Paulus D., Lacey G., Vilarino F. (2009). Texture-based polyp detection in colonoscopy. Inf. aktuell, 346–350. 10.1007/978-3-540-93860-6_70 DOI

Bardhi O., Sierra-Sosa D., Garcia-Zapirain B., Bujanda L. (2021). Deep Learning Models for Colorectal Polyps. Inf. 2021 12 (6), 245. 10.3390/INFO12060245 DOI

Bernal J., Sánchez F. J., Fernández-Esparrach G., Gil D., Rodríguez C., Vilariño F. (2015). WM-DOVA maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians. Comput. Med. imaging Graph. 43, 99–111. 10.1016/J.COMPMEDIMAG.2015.02.007 PubMed DOI

Bernal J., Sánchez J., Vilariño F. (2012). Towards automatic polyp detection with a polyp appearance model. Pattern Recognit. 45 (9), 3166–3182. 10.1016/J.PATCOG.2012.03.002 DOI

Bodenstedt S., Allan M., Agustinos A., Du X., Garcia-Peraza-Herrera L., Kenngott H., et al. (2018). Comparative evaluation of instrument segmentation and tracking methods in minimally invasive surgery. Available at: https://arxiv.org/abs/1805.02475v1 (Accessed December 18, 2023).

Bray F., Ferlay J., Soerjomataram I., Siegel R. L., Torre L. A., Jemal A. (2018). Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA a cancer J. Clin. 68 (6), 394–424. 10.3322/CAAC.21492 PubMed DOI

Chen L. C., Papandreou G., Kokkinos I., Murphy K., Yuille A. L. (2018). DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and fully Connected CRFs. IEEE Trans. Pattern Analysis Mach. Intell. 40 (4), 834–848. 10.1109/TPAMI.2017.2699184 PubMed DOI

Deng J., Dong W., Socher R., Jia-Li L., Li K., Fei-Fei L., et al. (2010). “ImageNet: A large-scale hierarchical image database,” in 2009 IEEE Conference on Computer Vision and Pattern Recognitionpp, Miami, FL, USA, 20-25 June 2009, 248–255. 10.1109/CVPR.2009.5206848 DOI

Doubeni C. A., Corley D. A., Quinn V. P., Jensen C. D., Zauber A. G., Goodman M., et al. (2018). Effectiveness of screening colonoscopy in reducing the risk of death from right and left colon cancer: a large community-based study. Gut 67 (2), 291–298. 10.1136/GUTJNL-2016-312712 PubMed DOI PMC

Drozdzal M., Vorontsov E., Chartrand G., Kadoury S., Pal C. (2016). “The importance of skip connections in biomedical image segmentation,” in Deep learning and data labeling for medical applications. DLMIA LABELS 2016. Lecture Notes in Computer Science. Editors Carneiro G. (Springer, Cham; ), 10008. 10.1007/978-3-319-46976-8_19 DOI

He K., Zhang X., Ren S., Sun J. (2016). “Deep residual learning for image recognition,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27-30 June 2016, 770–778. 10.1109/CVPR.2016.90 DOI

Hicks S. A., Jha D., Thambawita V., Halvorsen P., Hammer H. L., Riegler M. A. (2021). The EndoTect 2020 Challenge: Evaluation and Comparison of Classification, Segmentation and Inference Time for Endoscopy. Lect. Notes Comput. Sci. Incl. Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinforma. 12668 LNCS, 263–274. 10.1007/978-3-030-68793-9_18 DOI

Hou Q., Cheng M. M., Hu X., Borji A., Tu Z., Torr P. H. S. (2016). Deeply supervised salient object detection with short connections. IEEE Trans. Pattern Analysis Mach. Intell. 41 (4), 815–828. 10.1109/tpami.2018.2815688 PubMed DOI

Hu J., Shen L., Sun G. (2018). “Squeeze-and-Excitation Networks,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18-23 June 2018, 7132–7141. 10.1109/CVPR.2018.00745 DOI

Huang C.-H., Wu H.-Y., Lin Y.-L. (2021). HarDNet-MSEG: A Simple Encoder-Decoder Polyp Segmentation Neural Network that Achieves over 0.9 Mean Dice and 86 FPS. Available at: https://arxiv.org/abs/2101.07172v2 (Accessed December 18, 2023).

Huang H., Lin L., Tong R., Hu H., Zhang Q., Iwamoto Y., et al. (2020). “UNet 3+: A Full-Scale Connected UNet for Medical Image Segmentation,” in ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, 2020-May, 1055–1059. 10.1109/ICASSP40776.2020.9053405 DOI

Iandola F. N., Han S., Moskewicz M. W., Ashraf K., Dally W. J., Keutzer K. (2016). SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0. Available at: https://arxiv.org/abs/1602.07360v4 (Accessed December 18, 2023).

Ibtehaz N., Rahman M. S. (2020). MultiResUNet : Rethinking the U-Net architecture for multimodal biomedical image segmentation. Neural Netw. 121, 74–87. 10.1016/J.NEUNET.2019.08.025 PubMed DOI

Issa I. A., NouredDine M. (2017). Colorectal cancer screening: An updated review of the available options. World J. gastroenterology 23 (28), 5086–5096. 10.3748/WJG.V23.I28.5086 PubMed DOI PMC

Jemal A., Center M. M., DeSantis C., Ward E. M. (2010). Global patterns of cancer incidence and mortality rates and trends. Cancer Epidemiol. biomarkers Prev. 19 (8), 1893–1907. 10.1158/1055-9965.EPI-10-0437 PubMed DOI

Jha D., Ali S., Emanuelsen K., Hicks S. A., Thambawita V., Garcia-Ceja E., et al. (2020b). Kvasir-Instrument: Diagnostic and therapeutic tool segmentation dataset in gastrointestinal endoscopy. Lect. Notes Comput. Sci., 218–229. 10.1007/978-3-030-67835-7_19 DOI

Jha D., Tomar N. K., Sharma V., Bagci U. (2023). TransNetR: Transformer-based Residual Network for Polyp Segmentation with Multi-Center Out-of-Distribution Testing. Available at: https://arxiv.org/abs/2303.07428v1 (Accessed December 18, 2023).

Jha D., Riegler M. A., Johansen D., Halvorsen P., Johansen H. D. (2020a). “DoubleU-Net: A deep convolutional neural network for medical image segmentation,” in Proceedings - IEEE Symposium on Computer-Based Medical Systems, 2020-July, 558–564. 10.1109/CBMS49503.2020.00111 DOI

Jha D., Smedsrud P. H., Johansen D., de Lange T., Johansen H. D., Halvorsen P., et al. (2021b). A Comprehensive Study on Colorectal Polyp Segmentation With ResUNet++, Conditional Random Field and Test-Time Augmentation. IEEE J. Biomed. health Inf. 25 (6), 2029–2040. 10.1109/JBHI.2021.3049304 PubMed DOI

Jha D., Smedsrud P. H., Riegler M. A., Halvorsen P., de Lange T., Johansen D., et al. (2019a). Kvasir-SEG: A Segmented Polyp Dataset. Lect. Notes Comput. Sci., 451–462. 10.1007/978-3-030-37734-2_37 DOI

Jha D., Smedsrud P. H., Riegler M. A., Johansen D., Lange T. D., Halvorsen P., et al. (2019b). “ResUNet++: An Advanced Architecture for Medical Image Segmentation,” in Proceedings - 2019 IEEE International Symposium on Multimedia, ISM 2019, San Diego, CA, USA, 09-11 December 2019, 225–230. 10.1109/ISM46123.2019.00049 DOI

Jha D., Tomar N. K., Ali S., Riegler M. A., Johansen H. D., Johansen D., et al. (2021a). “NanoNet: Real-Time Polyp Segmentation in Video Capsule Endoscopy and Colonoscopy,” in Proceedings - IEEE Symposium on Computer-Based Medical Systems, 2021-June, Aveiro, Portugal, 07-09 June 2021, 37–43. 10.1109/CBMS52027.2021.00014 DOI

Joseph F. J. J., Nonsiri S., Monsakul A. (2021). Keras and TensorFlow: A Hands-On Experience. EAI/Springer Innovations Commun. Comput., 85–111. 10.1007/978-3-030-66519-7_4 DOI

Karkanis S. A., Iakovidis D., Maroulis D., Karras D., Tzivras M. (2003). Computer-aided tumor detection in endoscopic video using color wavelet features. IEEE Trans. Inf. Technol. Biomed. 7 (3), 141–152. 10.1109/TITB.2003.813794 PubMed DOI

Kim Y. D., Park E., Yoo S., Choi T., Yang L., Shin D. (2015). Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications. Available at: https://arxiv.org/abs/1511.06530v2 (Accessed December 18, 2023).

Lee J., Park S. W., Kim Y. S., Lee K. J., Sung H., Song P. H., et al. (2017). Risk factors of missed colorectal lesions after colonoscopy. Medicine 96 (27), e7468. 10.1097/MD.0000000000007468 PubMed DOI PMC

Lee J. Y., Jeong J., Song E. M., Ha C., Lee H. J., Koo J. E., et al. (2020). Real-time detection of colon polyps during colonoscopy using deep learning: systematic validation with four independent datasets. Sci. Rep. 10 (1), 8379. 10.1038/S41598-020-65387-1 PubMed DOI PMC

Li H., Xiong P., An J., Wang L. (2018). Pyramid Attention Network for Semantic Segmentation. Available at: https://arxiv.org/abs/1805.10180v3 (Accessed December 18, 2023).

Litjens G., Kooi T., Bejnordi B. E., Setio A. A. A., Ciompi F., Ghafoorian M., et al. (2017). A survey on deep learning in medical image analysis. Med. image Anal. 42, 60–88. 10.1016/J.MEDIA.2017.07.005 PubMed DOI

Milletari F., Navab N., Ahmadi S. A. (2016). “V-Net: Fully convolutional neural networks for volumetric medical image segmentation,” in Proceedings - 2016 4th International Conference on 3D Vision, Stanford, CA, USA, 25-28 October 2016, 565–571. 10.1109/3DV.2016.79 DOI

Moshkov N., Mathe B., Kertesz-Farkas A., Hollandi R., Horvath P. (2020). Test-time augmentation for deep learning-based cell segmentation on microscopy images. Sci. Rep. 10 (1), 5068. 10.1038/S41598-020-61808-3 PubMed DOI PMC

Powers D. M. W. Ailab (2020). Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation. Available at: https://arxiv.org/abs/2010.16061v1 (Accessed December 18, 2023).

Qin X., Zhang Z., Huang C., Dehghan M., Zaiane O. R., Jagersand M. (2020). U2-Net: Going deeper with nested U-structure for salient object detection. Pattern Recognit. 106, 107404. 10.1016/j.patcog.2020.107404 DOI

Ronneberger O., Fischer P., Brox T. (2015). U-net: Convolutional networks for biomedical image segmentation. Lect. Notes Comput. Sci. 9351, 234–241. 10.1007/978-3-319-24574-4_28 DOI

Sánchez F. J., Bernal J., Sánchez-Montes C., de Miguel C. R., Fernández-Esparrach G. (2017). Bright spot regions segmentation and classification for specular highlights detection in colonoscopy videos. Mach. Vis. Appl. 28 (8), 917–936. 10.1007/S00138-017-0864-0 DOI

Sandler M., Howard A., Zhu M., Zhmoginov A., Chen L. C. (2018). “MobileNetV2: Inverted Residuals and Linear Bottlenecks,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 4510–4520. 10.1109/CVPR.2018.00474 DOI

Shamir R. R., Duchin Y., Kim J., Sapiro G., Harel N. (2018). Continuous Dice Coefficient: a Method for Evaluating Probabilistic Segmentations. bioRxiv, 306977. 10.1101/306977 DOI

Srivastava A., Jha D., Chanda S., Pal U., Johansen H., Johansen D., et al. (2021). MSRF-Net: A Multi-Scale Residual Fusion Network for Biomedical Image Segmentation. IEEE J. Biomed. Health Inf. 26 (5), 2252–2263. 10.1109/JBHI.2021.3138024 PubMed DOI

Sun J., Darbehani F., Zaidi M., Wang B. (2020). SAUNet: Shape Attentive U-Net for Interpretable Medical Image Segmentation. Lect. Notes Comput. Sci., 797–806. 10.1007/978-3-030-59719-1_77 DOI

Tan H. H., Lim K. H. (2019). “Vanishing Gradient Mitigation with Deep Learning Neural Network Optimization,” in 2019 7th International Conference on Smart Computing and Communications (ICSCC), Sarawak, Malaysia, 28-30 June 2019. 10.1109/ICSCC.2019.8843652 DOI

Uraoka T., Hosoe N., Yahagi N. (2015). Colonoscopy: is it as effective as an advanced diagnostic tool for colorectal cancer screening?. Expert Rev. gastroenterology hepatology 9 (2), 129–132. 10.1586/17474124.2015.960397 PubMed DOI

Valanarasu J. M. J., Patel V. M. (2022). UNeXt: MLP-Based Rapid Medical Image Segmentation Network. Lect. Notes Comput. Sci., 23–33. 10.1007/978-3-031-16443-9_3 DOI

Wang G., Li W., Zuluaga M. A., Pratt R., Patel P. A., Aertsen M., et al. (2018). Interactive Medical Image Segmentation Using Deep Learning With Image-Specific Fine Tuning. Ieee Trans. Med. Imaging 37 (7), 1562–1573. 10.1109/TMI.2018.2791721 PubMed DOI PMC

Wang Y., Zhou Q., Liu J., Xiong J., Gao G., Wu X., et al. (2019). “LEDNet: A Lightweight Encoder-Decoder Network for Real-Time Semantic Segmentation,” in Proceedings - International Conference on Image Processing, 1860–1864. 10.1109/ICIP.2019.8803154 DOI

Xiang L., Zhan Q., Zhao X. H., Wang Y. D., An S. L., Xu Y. Z., et al. (2014). Risk factors associated with missed colorectal flat adenoma: A multicenter retrospective tandem colonoscopy study. World J. Gastroenterology WJG 20 (31), 10927. 10.3748/WJG.V20.I31.10927 PubMed DOI PMC

Yamada M., Saito Y., Imaoka H., Saiko M., Yamada S., Kondo H., et al. (2019). Development of a real-time endoscopic image diagnosis support system using deep learning technology in colonoscopy. Sci. Rep. 2019 9 (1), 14465–14469. 10.1038/s41598-019-50567-5 PubMed DOI PMC

Zhou Z., Rahman Siddiquee M. M., Tajbakhsh N., Liang J. (2018). UNet++: A Nested U-Net Architecture for Medical Image Segmentation. Lect. Notes Comput. Sci. 11045, 3–11. 10.1007/978-3-030-00889-5_1 PubMed DOI PMC

Zimmermann-Fraedrich K., Sehner S., Rex D. K., Kaltenbach T., Soetikno R., Wallace M., et al. (2019). Right-Sided Location Not Associated With Missed Colorectal Adenomas in an Individual-Level Reanalysis of Tandem Colonoscopy Studies. Gastroenterology 157 (3), 660–671.e2. 10.1053/J.GASTRO.2019.05.011 PubMed DOI