BACKGROUND: There is an increasing trend to represent domain knowledge in structured graphs, which provide efficient knowledge representations for many downstream tasks. Knowledge graphs are widely used to model prior knowledge in the form of nodes and edges to represent semantically connected knowledge entities, which several works have adopted into different medical imaging applications. METHODS: We systematically searched over five databases to find relevant articles that applied knowledge graphs to medical imaging analysis. After screening, evaluating, and reviewing the selected articles, we performed a systematic analysis. RESULTS: We looked at four applications in medical imaging analysis, including disease classification, disease localization and segmentation, report generation, and image retrieval. We also identified limitations of current work, such as the limited amount of available annotated data and weak generalizability to other tasks. We further identified the potential future directions according to the identified limitations, including employing semisupervised frameworks to alleviate the need for annotated data and exploring task-agnostic models to provide better generalizability. CONCLUSIONS: We hope that our article will provide the readers with aggregated documentation of the state-of-the-art knowledge graph applications for medical imaging to encourage future research.

Institute of Formal and Applied Linguistics Charles University Czechia Czech Republic

Population Health Sciences Weill Cornell Medicine New York USA

The University of Texas at Austin Austin USA

Zobrazit více v PubMed

Ji S., Pan S., Cambria E., Marttinen P., and Philip S. Y., “A survey on knowledge graphs: representation, acquisition, and applications,” IEEE Transactions on Neural Networks and Learning Systems, vol. 33, no. 2, pp. 494–514, 2022 PubMed

Auer S., Bizer C., Kobilarov G., Lehmann J., Cyganiak R., and Ives Z., “DBpedia: a nucleus for a web of open data,” in 6th International Semantic Web Conference, 2nd Asian Semantic Web Conference, ISWC 2007 + ASWC 2007, Busan, Korea, 2007,

Carlson A., Betteridge J., Kisiel B., Settles B., Hruschka Jr E. R., and Mitchell T. M., “Toward an architecture for never-ending language learning,” in Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, Atlanta, Georgia, USA, 2010,

Vrandečić D., and Krötzsch M., “Wikidata: a free collaborative knowledgebase,” Communications of the ACM, vol. 57, no. 10, pp. 78–85, 2014

Sumithra M. K., and Sridhar R., “Information retrieval in financial documents,” Evolving Technologies for Computing, Communication and Smart World., Springer, Singapore, 2020

Bastos A., Nadgeri A., Singh K., Mulang I. O., Shekarpour S., Hoffart J., and Kaul M., “RECON: relation extraction using knowledge graph context in a graph neural network,” in The World Wide Web Conference 2021, Ljubljana, Slovenia, 2021,

Fei H., Ren Y., Zhang Y., Ji D., and Liang X., “Enriching contextualized language model from knowledge graph for biomedical information extraction,” Briefings in Bioinformatics, vol. 22, no. 3, 2021 PubMed

Jaradeh M. Y., Singh K., Stocker M., Both A., and Auer S., “Better call the plumber: orchestrating dynamic information extraction pipelines,” in International Conference on Web Engineering, Biarritz, France, 2021,

Banerjee P., and Baral C., “Self-supervised knowledge triplet learning for zero-shot question answering,” 2020, https://arxiv.org/abs/2005.00316.

Ma K., Ilievski F., Francis J., Bisk Y., Nyberg E., and Oltramari A., “Knowledge-driven data construction for zero-shot evaluation in commonsense question answering,” in Proceedings of the 35th AAAI Conference on Artificial Intelligence, Virtual, 2021,

, “KERL: a knowledge-guided reinforcement learning model for sequential recommendation,” in ACM SIGIR Conference on Research and Development in Information Retrieval, Xi'an, China, 2020, Wang P., Fan Y., Xia L., Zhao W. X., Niu S. Z., and Huang J., Eds.,

Wang X., He X., Cao Y., Liu M., and Chua T.-S., “KGAT: knowledge graph attention network for recommendation,” in KDD ‘19: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Anchorage, AK, USA, 2019,

Xiang W., Huang T., Wang D., Yuan Y., Liu Z., He X., and Chua T.-S., “Learning intents behind interactions with knowledge graph for recommendation,” in Proceedings of the Web Conference 2021, Ljubljana, Slovenia, 2021,

Xi J., Ye L., Huang Q., and Li X., “Tolerating data missing in breast cancer diagnosis from clinical ultrasound reports via knowledge graph inference,” in KDD ‘21: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, Singapore, 2021,

Dai Y., Guo C., Guo W., and Eickhoff C., “Drug–drug interaction prediction with Wasserstein Adversarial Autoencoder-based knowledge graph embeddings,” Briefings in Bioinformatics, vol. 22, no. 4, 2021 PubMed

Yu Y., Huang K., Zhang C., Glass L. M., Sun J., and Xiao C., “SumGNN: multi-typed drug interaction prediction via efficient knowledge graph summarization,” Bioinformatics, vol. 37, no. 18, pp. 2988–2995, 2021 PubMed PMC

Chilińskiab M., Senguptab K., and Plewczynski D., “From DNA human sequence to the chromatin higher order organisation and its biological meaning: using biomolecular interaction networks to understand the influence of structural variation on spatial genome organisation and its functional effect,” Seminars in Cell & Developmental Biology, vol. 121, pp. 171–185, 2022 PubMed

Xie X., Niu J., Liu X., Chen Z., Tang S., and Shui Y., “A survey on incorporating domain knowledge into deep learning for medical image analysis,” Medical Image Analysis, vol. 69, article 101985, 2021 PubMed

Goodfellow I., Bengio Y., and Courville A.. Deep Learning, The MIT Press, 2016

Nicholson D. N., and Greene C. S., “Constructing knowledge graphs and their biomedical applications,” Computational and Structural Biotechnology Journal, vol. 18, pp. 1414–1428, 2020 PubMed PMC

Richens R. H., “Preprogramming for mechanical translation,” Mechanical Translation and Computational Linguistics, vol. 3, no. 1, pp. 20–25, 1956

Shortliffe E.Computer-based medical consultations, MYCIN, Elsevier, 1976

Miller E.An Introduction to the Resource Description Framework, D-lib Magazine, 2005

McGuinness D.OWL Web ontology language overview, W3C recommendation, 2004

Bizer C., Heath T., and Berners-Lee T., “Linked Data-The Story So Far,” Semantic services, interoperability and web applications: emerging concepts, IGI global, pp. 205–227, 2009

Miller G., “WordNet: a lexical database for English,” Communications of the ACM, vol. 38, no. 11, pp. 39–41, 1995

Suchanek F. M., Kasneci G., and Weikum G., “Yago: a core of semantic knowledge,” in Proceedings of the 16th international conference on World Wide Web, Banff, Alberta, Canada, 2007, pp. 697–706

Bollacker K., Evans C., Paritosh P., Sturge T., and Taylor J., “Freebase: a collaboratively created graph database for structuring human knowledge,” in Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, Vancouver, Canada, 2008,

Dong X., Gabrilovich E., Heitz G., Horn W., Lao N., Murphy K., Strohmann T., Sun S., and Zhang W., “Knowledge vault: a web-scale approach to probabilistic knowledge fusion,” in The 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, NY, USA, 2014,

Cui S., and Shrouty D., “Interest taxonomy: a knowledge graph management system for content understanding at Pinterest,” 2020

Dong X. L., He X., Kan A., Li X., Liang Y., Ma J., Xu Y. E., Zhang C., Zhao T., Blanco Saldana G., and Deshpande S., “AutoKnow: self-driving knowledge collection for products of thousands of types,” 2020, https://ui.adsabs.harvard.edu/abs/2020arXiv200613473L/abstract.

Färber M., “The Microsoft Academic Knowledge Graph: A Linked Data Source with 8 Billion Triples of Scholarly Data,” in Proceedings of the 18th International Semantic Web Conference, Auckland, New Zealand, 2019,

Chan M. F., Witztum A., and Valdes G., “Integration of AI and machine learning in radiotherapy QA,” Frontiers in Artificial Intelligence, vol. 3, 2020 PubMed PMC

, “Synergizing medical imaging and radiotherapy with deep learning,” Machine Learning: Science and Technology, Shan H., Jia X., Yan P., Li Y., Paganetti H., and Wang G., Eds., vol. 1, no. 2, 2020

Skripcak T., Belka C., Bosch W., Brink C., Brunner T., Budach V., Büttner D., Debus J., Dekker A., Grau C., Gulliford S., Hurkmans C., Just U., Krause M., Lambin P., Langendijk J. A., Lewensohn R., Lühr A., Maingon P., Masucci M., Niyazi M., Poortmans P., Simon M., Schmidberger H., Spezi E., Stuschke M., Valentini V., Verheij M., Whitfield G., Zackrisson B., Zips D., and Baumann M., “Creating a data exchange strategy for radiotherapy research: towards federated databases and anonymised public datasets,” Radiotherapy and Oncology, vol. 113, no. 3, pp. 303–309, 2014 PubMed PMC

Obaid K. B., Zeebaree S., and Ahmed O. M., “Deep learning models based on image classification: a review,” International Journal of Science and Business, vol. 4, no. 11, pp. 75–81, 2020

Demner-Fushman D., Kohli M. D., Rosenman M. B., Shooshan S. E., Rodriguez L., Antani S., Thoma G. R., and McDonald C. J., “Preparing a collection of radiology examinations for distribution and retrieval,” Journal of the American Medical Informatics Association., vol. 23, no. 2, pp. 304–310, 2016 PubMed PMC

Wang X., Peng Y., Lu L., Lu Z., Bagheri M., and Summer R. M., “ChestX-ray8: hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, Hawaii, USA, 2017, pp. 2097–2106

Irvin J., Rajpurkar P., Ko M., Yu Y., Ciurea-Ilcus S., Chute C., Marklund H., Haghgoo B., Ball R., and Shpanskaya K., “CheXpert: a large chest radiograph dataset with uncertainty labels and expert comparison,” in Proceedings of the AAAI conference on artificial intelligence, Honolulu, Hawaii, USA, 2019, vol. 33, no. 1, pp. 590–597

Xie Y., Xia Y., Zhang J., Song Y., Fang D., Fulham M., and Cai W., “Knowledge-based collaborative deep learning for benign-malignant lung nodule classification on chest CT,” IEEE Transactions on Medical Imaging, vol. 38, no. 4, pp. 991–1004, 2019 PubMed

Yu X., Wang S.-H., and Zhang Y.-D., “CGNet: a graph-knowledge embedded convolutional neural network for detection of pneumonia,” Information Processing and Management, vol. 58, no. 1, p. 102411, 2021 PubMed PMC

, “AMA-GCN: adaptive multi-layer aggregation graph convolutional network for disease prediction,” Chen H., Zhuang F.-Z., Xiao L., Ma L., Liu H., Zhang R., Jiang H., and He Q., Eds., 2021, https://arxiv.org/abs/2106.08732.

Liu Y., Zhang F., Chen C., Wang S., Wang Y., and Yizhou Y., “Act like a radiologist: towards reliable multi-view correspondence reasoning for mammogram mass detection,” IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), p. 1, 2021 PubMed

Fu X., Bi L., Kumar A., Fulham M., and Kim J., “Graph-based intercategory and intermodality network for multilabel classification and melanoma diagnosis of skin lesions in dermoscopy and clinical images,” 2021, https://arxiv.org/abs/2104.00201. PubMed

Zhang Y., Wang X., Ziyue X., Qihang Y., Yuille A., and Daguang X., “When radiology report generation meets knowledge graph,” in Proceedings of the AAAI Conference on Artificial Intelligence, New York, New York, USA, 2020, pp. 12910–12917

Hou D., Zhao Z., and Sanyuan H., “Multi-label learning with visual-semantic embedded knowledge graph for diagnosis of radiology imaging,” IEEE Access (IEEE), vol. 9, pp. 15720–15730, 2021

Zhou Y., Zhou T., Zhou T., Huazhu F., Liu J., and Shao L., “Contrast-attentive thoracic disease recognition with dual-weighting graph reasoning,” IEEE Transactions on Medical Imaging (IEEE), vol. 40, no. 4, pp. 1196–1206, 2021 PubMed

Agu N. N., Wu J. T., Chao H., Lourentzou I., Sharma A., Moradi M., Yan P., and Hendler J., “AnaXNet: anatomy aware multi-label finding classification in chest X-ray,” in Proceedings of the 24th International Conference on Medical Image Computing and Computer Assisted Intervention, Strasbourg, France, 2021,

Sekuboyina A., Oñoro-Rubio D., Kleesiek J., and Malone B., “A relational-learning perspective to multi-label chest X-ray classification,” in 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), Nice, France, 2021,

Chen B., Li J., Guangming L., Hongbing Y., and Zhang D., “Label co-occurrence learning with graph convolutional networks for multi-label chest X-ray image classification,” IEEE Journal of Biomedical and Health Informatics, vol. 24, no. 8, pp. 2292–2302, 2020 PubMed

Zheng W., Yan L., Gou C., Zhang Z.-C., Zhang J. J., Hu M., and Wang F.-Y., “Pay attention to doctor-patient dialogues: multi-modal knowledge graph attention image-text embedding for COVID-19 diagnosis,” Information Fusion, vol. 75, pp. 168–185, 2021 PubMed PMC

Mudiyanselage T. B., Senanayake N., Ji C., Pan Y., and Zhang Y., “Covid-19 detection from chest X-ray and patient metadata using graph convolutional neural networks,” 2021, https://arxiv.org/abs/2105.09720.

Armato I. I. I., Samuel G., McLennan G., Bidaut L., McNitt-Gray M. F., Meyer C. R., Reeves A. P., Zhao B., Aberle D. R., Henschke C. I., and Hoffman E. A., “The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): a completed reference database of lung nodules on CT scans,” Medical physics, vol. 38, no. 2, pp. 915–931, 2011 PubMed PMC

Yang X., He X., Zhao J., Zhang Y., Zhang S., and Xie P., “COVID-CT-Dataset: A CT Scan Dataset about COVID-19,” 2020, https://covid-19.conacyt.mx/jspui/handle/1000/4157.

Kermany D., Zhang K., and Goldbaum M.. Labeled optical coherence tomography (OCT) and chest X-ray images for classification, Mendeley data, 2018

Johnson A. E., Pollard T. J., Greenbaum N. R., Lungren M. P., Deng C. Y., Peng Y., Lu Z., Mark R. G., Berkowitz S. J., and Horng S., “MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs,” 2019, https://arxiv.org/abs/1901.07042. PubMed PMC

Wu J. T., Agu N. N., Lourentzou I., Sharma A., Paguio J. A., Yao J. S., Dee E. C., Mitchell W. G., Kashyap S., and Giovannini A., “Chest ImaGenome dataset for clinical reasoning,” 2021, https://arxiv.org/abs/2108.00316.

Martino A., Yan C.-G., Li Q., Denio E., Castellanos F. X., Alaerts K., Anderson J., Assaf M., Bookheimer S., and Dapretto M., “The autism brain imaging data exchange: towards a large-scale evaluation of the intrinsic brain architecture in autism,” Molecular Psychiatry, vol. 19, no. 6, pp. 659–667, 2014 PubMed PMC

Lee R. S., Gimenez F., Hoogi A., Miyake K. K., Gorovoy M., and Rubin D. L., “A curated mammography data set for use in computer-aided detection and diagnosis research,” Scientific data, vol. 4, no. 1, article 170177, 2017 PubMed PMC

Cohen J. P., Morrison P., and Dao L., “COVID-19 Image Data Collection,” 2020, https://arxiv.org/abs/2003.11597.

Chowdhury M. E. H., Rahman T., Khandakar A., Mazhar R., Kadir M. A., Mahbub Z. B., Islam K. R., Khan M. S., Al Iqbal A., and Emadi N., “Can AI help in screening Viral and COVID-19 pneumonia?,” IEEE Access, vol. 8, pp. 132665–132676, 2020

Rahman T., Khandakar A., Qiblawey Y., Tahir A., Kiranyaz S., Kashem S. B. A., Al Islam M. T., Maadeed S., Zughaier S. M., and Khan M. S., “Exploring the effect of image enhancement techniques on COVID-19 detection using chest X-ray images,” Computers in Biology and Medicine, vol. 132, article 104319, 2021 PubMed PMC

Kawahara J., Daneshvar S., Argenziano G., and Hamarneh G., “Seven-point checklist and skin lesion classification using multitask multimodal neural nets,” IEEE Journal of Biomedical and Health Informatics, vol. 23, no. 2, pp. 538–546, 2019 PubMed

Sharma N., and Aggarwal L. M., “Automated medical image segmentation techniques,” Journal of Medical Physics, vol. 35, no. 1, pp. 3–14, 2010 PubMed PMC

Qi B., Zhao G., Wei X., Fang C., Pan C., Li J., He H., and Jiao L., “GREN: graph-regularized embedding network for weakly-supervised disease localization in X-ray images,” 2021, https://arxiv.org/abs/2107.06442. PubMed

Peng Y., Zhong H., Zheng X., Hongbin T., Li X., and Peng L., “Pulmonary lobe segmentation in CT images based on lung anatomy knowledge,” Mathematical Problems in Engineering, vol. 2021, –15, 2021

Zhao G., Qi B., and Li J., “Cross chest graph for disease diagnosis with structural relational reasoning,” in Proceedings of the 29th ACM International Conference on Multimedia, Chengdu, China, 2021,

Lassen B., van Rikxoort E. M., Schmidt M., Kerkstra S., van Ginneken B., and Kuhnigk J.-M., “Automatic segmentation of the lungs and lobes from thoracic CT scans,” 2011 PubMed

Ronneberger O., Fischer P., and Brox T., “U-Net: convolutional networks for biomedical image segmentation,” in International Conference on Medical image computing and computer-assisted intervention, Munich, Germany, 2015,

Li S., Tao Z., Li K., and Yun F., “Visual to text: survey of image and video captioning,” IEEE Transactions on Emerging Topics in Computational, vol. 3, no. 4, pp. 297–312, 2019

Li M., Wang F., Chang X., and Liang X., “Auxiliary signal-guided knowledge encoder-decoder for medical report generation,” 2020, https://arxiv.org/abs/2006.03744. PubMed PMC

Yuan J., Liao H., Luo R., and Luo J., “Automatic radiology report generation based on multi-view image fusion and medical concept enrichment,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, Shenzhen, China, 2019,

Li C., Liang X., Zhiting H., and Xing E., “Knowledge-driven encode, retrieve, paraphrase for medical image report generation,” in Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, Hawaii, USA, 2019a, pp. 6666–6673

Liu F., Wu X., Shen G., Fan W., and Zou Y., “Exploring and distilling posterior and prior knowledge for radiology report generation,” in Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR2021), Virtual, 2021a,

Radford A., Narasimhan K., Salimans T., and Sutskever I., “Improving language understanding by generative pre-training,” 2018

Qayyum A., Anwar S. M., Awais M., and Majid M., “Medical image retrieval using deep convolutional neural network,” Nerucomputing, vol. 266, pp. 8–20, 2017

Hwang K. H., Lee H., and Choi D., “Medical image retrieval: past and present,” Health Informatics Research, vol. 18, no. 1, pp. 3–9, 2012 PubMed PMC

Putzu L., Piras L., and Giacinto G., “Convolutional neural networks for relevance feedback in content based image retrieval,” Multimedia Tools and Applications, vol. 79, no. 37-38, pp. 26995–27021, 2020

Lacoste C., Chevallet J.-P., Lim J.-H., Wei X., Racoceanu D., Le D. T. H., Teodorescu R., and Vuillemenot N., “IPAL knowledge-based medical image retrieval in ImageCLEFmed 2006,” in 7th Workshop of the Cross-Language Evaluation Forum, Alicante, Spain, 2006,

Racoceanu D., Lacoste C., Teodorescu R., and Vuillemenot N., “A semantic fusion approach between medical images and reports using UMLS,” Information Retrieval Technology, vol. 4182, pp. 460–475, 2006