Computer Tomography (CT) is an imaging procedure that combines many X-ray measurements taken from different angles. The segmentation of areas in the CT images provides a valuable aid to physicians and radiologists in order to better provide a patient diagnose. The CT scans of a body torso usually include different neighboring internal body organs. Deep learning has become the state-of-the-art in medical image segmentation. For such techniques, in order to perform a successful segmentation, it is of great importance that the network learns to focus on the organ of interest and surrounding structures and also that the network can detect target regions of different sizes. In this paper, we propose the extension of a popular deep learning methodology, Convolutional Neural Networks (CNN), by including deep supervision and attention gates. Our experimental evaluation shows that the inclusion of attention and deep supervision results in consistent improvement of the tumor prediction accuracy across the different datasets and training sizes while adding minimal computational overhead.

Artificial Intelligence Laboratory Faculty of Applied Informatics Tomas Bata University in Zlin Zlin Czechia

Intelligent and Interactive Systems Department of Computer Science University of Innsbruck Innsbruck Austria

See more in PubMed

Bilic P., Christ P. F., Vorontsov E., Chlebus G., Chen H., Dou Q., et al. (2019). The liver tumor segmentation benchmark (lits). CoRR, abs/1901.04056.

Chen H., Dou Q., Yu L., Qin J., Heng P.-A. (2018). VoxResNet: Deep voxelwise residual networks for brain segmentation from 3D MR images. Neuroimage 170, 446–455. 10.1016/j.neuroimage.2017.04.041 PubMed DOI

Chen L.-C., Yang Y., Wang J., Xu W., Yuille A. L. (2016). “Attention to scale: scale-aware semantic image segmentation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Las Vegas, NV: ), 3640–3649. 10.1109/CVPR.2016.396 DOI

Corbetta M., Shulman G. L. (2002). Control of goal-directed and stimulus-driven attention in the brain. Nat. Rev. Neurosci. 3:201. 10.1038/nrn755 PubMed DOI

Dou Q., Chen H., Jin Y., Yu L., Qin J., Heng P.-A. (2016). “3D deeply supervised network for automatic liver segmentation from ct volumes,” in International Conference on Medical Image Computing and Computer-Assisted Intervention (Cham: Springer; ), 149–157. 10.1007/978-3-319-46723-8_18 DOI

Fukushima K. (1980). Neocognitron: a self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biol. Cybern. 36, 193–202. 10.1007/BF00344251 PubMed DOI

Gibson E., Giganti F., Hu Y., Bonmati E., Bandula S., Gurusamy K., et al. . (2018). Automatic multi-organ segmentation on abdominal ct with dense v-networks. IEEE Trans. Med. Imaging 37, 1822–1834. 10.1109/TMI.2018.2806309 PubMed DOI PMC

Grewal M., Srivastava M. M., Kumar P., Varadarajan S. (2018). “Radnet: Radiologist level accuracy using deep learning for hemorrhage detection in CT scans,” in 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018) (Washington, DC: ), 281–284. 10.1109/ISBI.2018.8363574 DOI

Hamidian S., Sahiner B., Petrick N., Pezeshk A. (2017). “3D convolutional neural network for automatic detection of lung nodules in chest CT,” in Medical Imaging 2017: Computer-Aided Diagnosis, Vol. 10134 (Orlando, FL: International Society for Optics and Photonics; ), 1013409 10.1117/12.2255795 PubMed DOI PMC

He K., Zhang X., Ren S., Sun J. (2015). “Delving deep into rectifiers: surpassing human-level performance on imagenet classification,” in IEEE International Conference on Computer Vision (Santiago: ), 1026–1034. 10.1109/ICCV.2015.123 DOI

Heller N., Sathianathen N., Kalapara A., Walczak E., Moore K., Kaluzniak H., et al. (2019). The kits19 challenge data: 300 kidney tumor cases with clinical context, CT semantic segmentations, and surgical outcomes. arXiv [preprint]. arXiv:1904.00445.

Hou X., Xie C., Li F., Nan Y. (2019). Cascaded Semantic Segmentation for Kidney and Tumor. Technical report, PingAn Technology Co., Ltd, Shanghai: 10.24926/548719.002 DOI

Hu P., Wu F., Peng J., Bao Y., Chen F., Kong D. (2017). Automatic abdominal multi-organ segmentation using deep convolutional neural network and time-implicit level sets. Int. J. Comput. Assist. Radiol. Surg. 12, 399–411. 10.1007/s11548-016-1501-5 PubMed DOI

Huang Z., Wang X., Huang L., Huang C., Wei Y., Liu W. (2019). “CCNet: Criss-cross attention for semantic segmentation,” in The IEEE International Conference on Computer Vision (ICCV) (Seoul: ). 10.1109/ICCV.2019.00069 DOI

Hubel D. H., Wiesel T. N. (1968). Receptive fields and functional architecture of monkey striate cortex. J. Physiol. 195, 215–243. 10.1113/jphysiol.1968.sp008455 PubMed DOI PMC

Isensee F., Kickingereder P., Wick W., Bendszus M., Maier-Hein K. H. (2017). “Brain tumor segmentation and radiomics survival prediction: contribution to the brats 2017 challenge,” in International MICCAI Brainlesion Workshop (Cham: Springer; ), 287–297. 10.1007/978-3-319-75238-9_25 DOI

Isensee F., Maier-Hein K. H. (2019). An Attempt at Beating the 3D U-Net. Technical report, German Cancer Research Center (DKFZ), Heidelberg: 10.24926/548719.001 DOI

Isensee F., Petersen J., Klein A., Zimmerer D., Jaeger P. F., Kohl S., et al. (2018). nnU-Net: Self-adapting framework for U-net-based medical image segmentation. CoRR abs/1809.10486. 10.1007/978-3-658-25326-4_7 DOI

Jimenez-del-Toro O., Müller H., Krenn M., Gruenberg K., Taha A. A., Winterstein M., et al. . (2016). Cloud-based evaluation of anatomical structure segmentation and landmark detection algorithms: visceral anatomy benchmarks. IEEE Trans. Med. Imaging 35, 2459–2475. 10.1109/TMI.2016.2578680 PubMed DOI

Kayalibay B., Jensen G., van der Smagt P. (2017). CNN-based segmentation of medical imaging data. CoRR abs/1701.03056.

Kearney V., Chan J. W., Wang T., Perry A., Yom S. S., Solberg T. D. (2019). Attention-enabled 3D boosted convolutional neural networks for semantic CT segmentation using deep supervision. Phys. Med. Biol. 64:135001. 10.1088/1361-6560/ab2818 PubMed DOI

Lee C.-Y., Xie S., Gallagher P., Zhang Z., Tu Z. (2015). “Deeply-supervised nets,” in Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, Vol. 38 of Proceedings of Machine Learning Research, eds G. Lebanon and S. V. N. Vishwanathan (San Diego, CA: PMLR; ), 562–570.

Linguraru M. G., Richbourg W. J., Liu J., Watt J. M., Pamulapati V., Wang S., et al. . (2012). Tumor burden analysis on computed tomography by automated liver and tumor segmentation. IEEE Trans. Med. Imaging 31, 1965–1976. 10.1109/TMI.2012.2211887 PubMed DOI PMC

Long J., Shelhamer E., Darrell T. (2015). “Fully convolutional networks for semantic segmentation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Boston, MA: ), 3431–3440. 10.1109/CVPR.2015.7298965 PubMed DOI

Milletari F., Navab N., Ahmadi S.-A. (2016). “V-Net: fully convolutional neural networks for volumetric medical image segmentation,” in 4th IEEE International Conference on 3D Vision (3DV) (Stanford, CA: Stanford University; ), 565–571. 10.1109/3DV.2016.79 DOI

Mnih V., Heess N., Graves A., Kavukcuoglu K. (2014). “Recurrent models of visual attention,” in Advances in Neural Information Processing Systems (Montreal, QC: ), 2204–2212.

Mu G., Lin Z., Han M., Yao G., Gao1 Y. (2019). Segmentation of Kidney Tumor by Multi-Resolution VB-nets. Technical report, Shanghai United Imaging Intelligence Inc, Shanghai. 10.24926/548719.003 DOI

Oktay O., Schlemper J., Folgoc L. L., Lee M. C. H., Heinrich M. P., Misawa K., et al. (2018). Attention U-net: Learning where to look for the pancreas. CoRR abs/1804.03999.

Rodríguez-Sánchez A. J., Fallah M., Leonardis A. (2015). Hierarchical object representations in the visual cortex and computer vision. Front. Comput. Neurosci. 9:142. 10.3389/fncom.2015.00142 PubMed DOI PMC

Ronneberger O., Fischer P., Brox T. (2015). “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention (Cham: Springer; ), 234–241. 10.1007/978-3-319-24574-4_28 DOI

Roth H. R., Lu L., Lay N., Harrison A. P., Farag A., Sohn A., et al. . (2018). Spatial aggregation of holistically-nested convolutional neural networks for automated pancreas localization and segmentation. Med. Image Anal. 45, 94–107. 10.1016/j.media.2018.01.006 PubMed DOI

Sharma N. K., Aggarwal L. M. (2010). Automated medical image segmentation techniques. J. Med. Phys. 35, 3–14. 10.4103/0971-6203.58777 PubMed DOI PMC

Simpson A. L., Antonelli M., Bakas S., Bilello M., Farahani K., van Ginneken B., et al. (2019). A large annotated medical image dataset for the development and evaluation of segmentation algorithms. CoRR, abs/1902.09063.

Wang F., Jiang M., Qian C., Yang S., Li C., Zhang H., et al. (2017). “Residual attention network for image classification,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (Honolulu, HI: ). 10.1109/CVPR.2017.683 DOI

Winzeck S., Hakim A., McKinley R., Pinto J. A., Alves V., Silva C., et al. . (2018). Isles 2016 and 2017-benchmarking ischemic stroke lesion outcome prediction based on multispectral MRI. Front. Neurol. 9:679. 10.3389/fneur.2018.00679 PubMed DOI PMC

Xiao T., Xu Y., Yang K., Zhang J., Peng Y., Zhang Z. (2015). “The application of two-level attention models in deep convolutional neural network for fine-grained image classification,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Boston, MA: ), 842–850.

Xu K., Ba J., Kiros R., Cho K., Courville A., Salakhudinov R., Zemel R., et al. (2015). “Show, attend and tell: Neural image caption generation with visual attention,” in International Conference on Machine Learning (Lille: ), 2048–2057.

Yang G., Li G., Pan T., Kong Y., Wu J., Shu H., et al. (2018). “Automatic segmentation of kidney and renal tumor in ct images based on 3d fully convolutional neural network with pyramid pooling module,” in 2018 24th International Conference on Pattern Recognition (ICPR), 3790–3795. 10.1109/ICPR.2018.8545143 DOI

Ypsilantis P.-P., Montana G. (2017). Learning what to look in chest x-rays with a recurrent visual attention model. arXiv [preprint]. arXiv:1701.06452.

Zhou Y., Xie L., Fishman E. K., Yuille A. L. (2017). “Deep supervision for pancreatic cyst segmentation in abdominal CT scans,” in International Conference on Medical Image Computing and Computer-Assisted Intervention (Cham: Springer; ), 222–230. 10.1007/978-3-319-66179-7_26 DOI

Zhu Q., Du B., Turkbey B., Choyke P. L., Yan P. (2017). “Deeply-supervised CNN for prostate segmentation,” in 2017 International Joint Conference on Neural Networks (IJCNN) (Anchorage, AK: ), 178–184. 10.1109/IJCNN.2017.7965852 DOI

Zreik M., Leiner T., de Vos B. D., van Hamersvelt R. W., Viergever M. A., Išgum I. (2016). “Automatic segmentation of the left ventricle in cardiac ct angiography using convolutional neural networks,” in 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI) (Prague, CZ: ), 40–43. 10.1109/ISBI.2016.7493206 DOI