PURPOSE: K trans $$ {K}^{\mathrm{trans}} $$ has often been proposed as a quantitative imaging biomarker for diagnosis, prognosis, and treatment response assessment for various tumors. None of the many software tools for K trans $$ {K}^{\mathrm{trans}} $$ quantification are standardized. The ISMRM Open Science Initiative for Perfusion Imaging-Dynamic Contrast-Enhanced (OSIPI-DCE) challenge was designed to benchmark methods to better help the efforts to standardize K trans $$ {K}^{\mathrm{trans}} $$ measurement. METHODS: A framework was created to evaluate K trans $$ {K}^{\mathrm{trans}} $$ values produced by DCE-MRI analysis pipelines to enable benchmarking. The perfusion MRI community was invited to apply their pipelines for K trans $$ {K}^{\mathrm{trans}} $$ quantification in glioblastoma from clinical and synthetic patients. Submissions were required to include the entrants' K trans $$ {K}^{\mathrm{trans}} $$ values, the applied software, and a standard operating procedure. These were evaluated using the proposed OSIP I gold $$ \mathrm{OSIP}{\mathrm{I}}_{\mathrm{gold}} $$ score defined with accuracy, repeatability, and reproducibility components. RESULTS: Across the 10 received submissions, the OSIP I gold $$ \mathrm{OSIP}{\mathrm{I}}_{\mathrm{gold}} $$ score ranged from 28% to 78% with a 59% median. The accuracy, repeatability, and reproducibility scores ranged from 0.54 to 0.92, 0.64 to 0.86, and 0.65 to 1.00, respectively (0-1 = lowest-highest). Manual arterial input function selection markedly affected the reproducibility and showed greater variability in K trans $$ {K}^{\mathrm{trans}} $$ analysis than automated methods. Furthermore, provision of a detailed standard operating procedure was critical for higher reproducibility. CONCLUSIONS: This study reports results from the OSIPI-DCE challenge and highlights the high inter-software variability within K trans $$ {K}^{\mathrm{trans}} $$ estimation, providing a framework for ongoing benchmarking against the scores presented. Through this challenge, the participating teams were ranked based on the performance of their software tools in the particular setting of this challenge. In a real-world clinical setting, many of these tools may perform differently with different benchmarking methodology.

Advanced Imaging Research Center Oregon Health and Science Institute Portland Oregon USA

Athinoula A Martinos Center for Biomedical Imaging Harvard Medical School Boston Massachusetts USA

Biomedical Imaging Center Livestrong Cancer Institutes University of Texas at Austin Austin Texas USA

Bioxydyn Ltd Manchester UK

Cancer Center Amsterdam Imaging and Biomarkers Amsterdam The Netherlands

Center for Biomedical Engineering Indian Institute of Technology Delhi New Delhi India

Center for Computational Imaging and Simulation Technologies in Biomedicine School of Computing School of Medicine University of Leeds Leeds UK

Center for Data Driven Discovery Division of Neurosurgery Children's Hospital of Philadelphia Philadelphia Pennsylvania USA

Center for Medical Image Computing Department of Medical Physics and Biomedical Engineering University College London London UK

Clinical Imaging Group Genentech Inc South San Francisco California USA

Corewell Health William Beaumont University Hospital Royal Oak Michigan USA

Czech Academy of Sciences Institute of Information Theory and Automation Praha Czech Republic

Czech Academy of Sciences Institute of Scientific Instruments Brno Czech Republic

Department of Biomedicine and Prevention University of Rome Tor Vergata Italy

Department of Computer Science University College London London UK

Department of Diagnostic Medicine University of Texas Austin Texas USA

Department of Imaging Physics MD Anderson Cancer Center Houston Texas USA

Department of Infection Immunity and Cardiovascular Disease University of Sheffield Sheffield UK

Department of Medical Physics Memorial Sloan Kettering Cancer Center New York New York USA

Department of Neurosurgery Perelman School of Medicine University of Pennsylvania Pennsylvania USA

Department of Radiological Sciences University of California Los Angeles California USA

Department of Radiology and Nuclear Medicine Erasmus MC University Medical Center Rotterdam The Netherlands

Department of Radiology and Nuclear Medicine University of Amsterdam Amsterdam The Netherlands

Department of Radiology Grossman School of Medicine New York University New York New York USA

Department of Radiology Medical College of Wisconsin Milwaukee Wisconsin USA

Department of Radiology Memorial Sloan Kettering Cancer Center New York New York USA

Department of Radiology Neuroradiology Division Mayo Clinic Scottsdale Arizona USA

Department of Radiology Perelman School of Medicine University of Pennsylvania Philadelphia Pennsylvania USA

Department of Radiology The Christie Hospital NHS Trust Manchester UK

Department of Radiology University of Alabama Birmingham Alabama USA

Department of Radiology University of Washington Seattle Washington USA

Department of Radiology University of Wisconsin Madison Madison Wisconsin USA

Department of Radiology Weill Cornell Medical College New York New York USA

Department of Surgery and Cancer Imperial College London UK

Department of Translational Neuroscience Barrow Neurological Institute Phoenix Arizona USA

Departments of Biomedical Engineering Diagnostic Medicine Oncology Livestrong Cancer Institutes Oden Institute for Computational Engineering and Sciences The University of Texas Austin Texas USA

Division of Cancer Sciences University of Manchester Manchester UK

Division of Radiotherapy and Imaging The Institute of Cancer Research London UK

Fraunhofer Institute for Digital Medicine MEVIS Bremen Germany

Institute of Bioengineering and Bioimaging Singapore Singapore

Institute of Psychiatry Psychology and Neuroscience King's College London UK

Lysholm Department of Neuroradiology National Hospital for Neurology and Neurosurgery University College London Hospitals NHS Foundation Trust London UK

Neuroradiology Division Department of Radiology Mayo Clinic Phoenix Arizona USA

Oden Institute for Computational Engineering and Sciences The University of Texas Austin Texas USA

Quantitative MR Imaging and Spectroscopy Group Research Center for Molecular and Cellular Imaging Tehran University of Medical Sciences Tehran Iran

School of Physics and Astronomy University of Leeds Leeds UK

University Medical Center Göttingen Göttingen Germany

See more in PubMed

Bell L, Ragunathan S, Kazerooni F. Contrast agent-based perfusion MRI methods. In: Choi IY, Jezzard P, eds. Advanced Neuro MR Techniques and Applications. Vol 4. Academic Press; 2021:195-209. doi:10.1016/B978-0-12-822479-3.00024-5

Zhang N, Zhang L, Qiu B, Meng L, Wang X, Hou B. Correlation of volume transfer coefficient Ktrans with histopathologic grades of gliomas. J Magn Reson Imaging. 2012;36:355-363. doi:10.1002/jmri.23675

Kickingereder P, Wiestler B, Graf M, et al. Evaluation of dynamic contrast-enhanced MRI derived microvascular permeability in recurrent glioblastoma treated with bevacizumab. J Neurooncol. 2015;121:373-380. doi:10.1007/s11060-014-1644-6

Thomas A, Arevalo-Perez J, Kaley T, et al. Dynamic contrast enhanced T1 MRI perfusion differentiates pseudoprogression from recurrent glioblastoma. J Neurooncol. 2015;125:183-190. doi:10.1007/s11060-015-1893-z

Zhu X, Li K, Jackson A. Dynamic contrast-enhanced MRI in cerebral tumours. In: Alan J, Buckley DL, Parker GJ, eds. Dynamic Contrast-Enhanced Magnetic Resonance Imaging in Oncology. Springer; 2005:117-143. doi:10.1007/3-540-26420-5_9

O'Connor J, Jackson A, Parker G, Jayson G. DCE-MRI biomarkers in the clinical evaluation of antiangiogenic and vascular disrupting agents. Br J Cancer. 2007;96:189-195. doi:10.1038/sj.bjc.6603515

QIBA MR Biomarker Committee. MR DCE-MRI Quantification (DCEMRI-Q), Quantitative Imaging Biomarkers Alliance. Profile Stage: Public Comment. http://qibawiki.rsna.org/index.php/Profiles (accessed June 2022)

Shukla-Dave A, Obuchowski NA, Chenevert TL, et al. Quantitative Imaging Biomarkers Alliance (QIBA) recommendations for improved precision of DWI and DCE-MRI derived biomarkers in multicenter oncology trials. J Magn Reson Imaging. 2019;49:e101-e121. doi:10.1002/jmri.26518

Lindquist M. Neuroimaging results altered by varying analysis pipelines. Nature. 2020;582:36-37. doi:10.1038/d41586-020-01282-z

Stikov N, Trzasko J, Bernstein M. Reproducibility and the future of MRI research. Magn Reson Med. 2019;82:1981-1983. doi:10.1002/mrm.27939

Simmons J, Nelson L, Simonsohn U. False-positive psychology. Psychol Sci. 2011;22:1359-1366. doi:10.1177/0956797611417632

Wicherts J, Veldkamp C, Augusteijn H, Bakker M, van Aert R, van Assen M. Degrees of freedom in planning, running, analyzing, and reporting psychological studies: a checklist to avoid p-hacking. Front Psychol. 2016;7:1832. doi:10.3389/fpsyg.2016.01832

OSIPI TF 1.2. Public Version of OSIPI_DSCDCEinventory: DSC+DCE Software. https://docs.google.com/spreadsheets/d/e/2PACX-1vSOHrNIiWcwDD5BoHij1dpXKgEejtohqKF6KZQMZi3G6GzM_BP8xpuwRbjFFvDW9Q/pubhtml?gid=1031101549&single=true (accessed June 2023).

Carp J. The secret lives of experiments: methods reporting in the fMRI literature. Neuroimage. 2012;63:289-300. doi:10.1016/j.neuroimage.2012.07.004

Kazerooni A, Bell L, Van den Abeele F, et al. The open source initiative for perfusion imaging (OSIPI): DCE-MRI challenge. In: Proceedings of the 10th Annual Meeting of ISMRM. 2021:1094.

Clark K, Vendt B, Smith K, et al. The cancer imaging archive (TCIA): maintaining and operating a public information repository. J Digit Imaging. 2013;26:1045-1057. doi:10.1007/s10278-013-9622-7

Barboriak D. Data From RIDER NEURO MRI. Cancer Imag Arch. 2015. https://doi.org/10.7937/K9/TCIA.2015.VOSN3HN1

Kazerooni A, Shalom E, Ahmed Z, et al. OSIPI DCE Challenge. Accessed August 2023. https://osf.io/u7a6f/

Murase K. Efficient method for calculating kinetic parameters usingT1-weighted dynamic contrast-enhanced magnetic resonance imaging. Magn Reson Med. 2004;51:858-862. doi:10.1002/mrm.20022

Brix G, Kiessling F, Lucht R, et al. Microcirculation and microvasculature in breast tumors: pharmacokinetic analysis of dynamic MR image series. Magn Reson Med. 2004;52:420-429. doi:10.1002/mrm.20161

Parker G, Roberts C, Macdonald A, et al. Experimentally-derived functional form for a population-averaged high-temporal-resolution arterial input function for dynamic contrast-enhanced MRI. Magn Reson Med. 2006;56:993-1000. doi:10.1002/mrm.21066

Pintaske J, Martirosian P, Graf H, et al. Relaxivity of Gadopentetate Dimeglumine (Magnevist), Gadobutrol (Gadovist), and Gadobenate Dimeglumine (MultiHance) in human blood plasma at 0.2, 1.5, and 3 Tesla. Invest Radiol. 2006;41:213-221. doi:10.1097/01.rli.0000197668.44926.f7

Siemonsen S, Finsterbusch J, Matschke J, Lorenzen A, Ding X, Fiehler J. Age-dependent normal values of T2* and T2′ in brain parenchyma. Am J Neuroradiol. 2008;29:950-955. doi:10.3174/ajnr.A0951

Zhang X, Petersen ET, Ghariq E, et al. In vivo blood T1 measurements at 1.5 T, 3 T, and 7 T. Magn Reson Med. 2013;70:1082-1086. doi:10.1002/mrm.24550

Yushkevich P, Gao Y, Gerig G. ITK-SNAP: an interactive tool for semi-automatic segmentation of multi-modality biomedical images. Proceedings of the 38th annual International Conference of the IEEE Engineering in Medicine and Biology Society; IEEE; 2016:3342-3345. doi:10.1109/EMBC.2016.7591443

Brett M, Markiewicz C, Hanke M, et al. nipy/nibabel: 3.2.1. 2020. Accessed March 2023. https://zenodo.org/records/4295521

Harris C, Millman K, van der Walt S, et al. Array programming with NumPy. Nature. 2020;585:357-362. doi:10.1038/s41586-020-2649-2

Matejka J, Fitzmaurice G. Same stats, different graphs. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM; 2017:1290-1294. doi:10.1145/3025453.3025912

Obuchowski N, Bullen J. Quantitative imaging biomarkers: effect of sample size and bias on confidence interval coverage. Stat Methods Med Res. 2018;27:3139-3150. doi:10.1177/0962280217693662

Obuchowski NA, Huang E, de Souza NM, et al. A framework for evaluating the technical performance of multiparameter quantitative imaging biomarkers (mp-QIBs). Acad Radiol. 2023;30:147-158. doi:10.1016/j.acra.2022.08.031

Ottens T, Barbieri S, Orton MR, et al. Deep learning DCE-MRI parameter estimation: application in pancreatic cancer. Med Image Anal. 2022;80:102512. doi:10.1016/j.media.2022.102512

Gurney-Champion OJ, Ottens T. DCE-NET, GitHu. https://github.com/oliverchampion/DCENET (accessed July 2022)

Berks M, Parker G, Little R, Cheung S. Madym: a C++ toolkit for quantitative DCE-MRI analysis. J Open Source Softw. 2021;6:3523. doi:10.21105/joss.03523

Berks M. madym_cxx. GitLab. https://gitlab.com/manchester_qbi/manchester_qbi_public/osipi-dce-challenge (accessed July 2022)

Jiřík R. PerfLab. CERIT Scientific Cloud. http://perflab.cerit-sc.cz/ (accessed July 14, 2022).

Jiřík R, Taxt T, Macíček O, et al. Blind deconvolution estimation of an arterial input function for small animal DCE-MRI. Magn Reson Imaging. 2019;62:46-56. doi:10.1016/j.mri.2019.05.024

Jiřík R, Bartoš M, Macíček O, Starčuk Z. Arterial input function estimation using all-channel blind deconvolution with spatial regularization in DCE-MRI. In: Proceedings of the 31st Annual Meeting of ISMRM. London, UK 2022 p. 1582.

Mikheev A, Rusinek H. FireVoxel. https://firevoxel.org/ (accessed July 2022).

Barnes SR, Ng TSC, Santa-Maria N, Montagne A, Zlokovic BV, Jacobs RE. ROCKETSHIP: a flexible and modular software tool for the planning, processing and analysis of dynamic MRI studies. BMC Med Imaging. 2015;15:19. doi:10.1186/s12880-015-0062-3

Schneider CA, Rasband WS, Eliceiri KW. NIH image to ImageJ: 25 years of image analysis. Nat Methods. 2012;9:671-675. doi:10.1038/nmeth.2089

Singh A, Rathore RKS, Haris M, Verma SK, Husain N, Gupta RK. Improved bolus arrival time and arterial input function estimation for tracer kinetic analysis in DCE-MRI. J Magn Reson Imaging. 2009;29:166-176. doi:10.1002/jmri.21624

Rata M, Collins DJ, Darcy J, et al. Assessment of repeatability and treatment response in early phase clinical trials using DCE-MRI: comparison of parametric analysis using MR- and CT-derived arterial input functions. Eur Radiol. 2016;26:1991-1998. doi:10.1007/s00330-015-4012-9

Port RE, Knopp MV, Brix G. Dynamic contrast-enhanced MRI using Gd-DTPA: interindividual variability of the arterial input function and consequences for the assessment of kinetics in tumors. Magn Reson Med. 2001;45:1030-1038. doi:10.1002/mrm.1137

Larsson H, Courivaud F, Rostrup E, Hansen A. Measurement of brain perfusion, blood volume, and blood-brain barrier permeability, using dynamic contrast-enhanced T1 weighted MRI at 3 Tesla. Magn Reson Med. 2009;62:1270-1281. doi:10.1002/mrm.22136

Taheri S, Gasparovic C, Huisa B, et al. Blood-brain barrier permeability abnormalities in vascular cognitive impairment. Stroke. 2011;42:2158-2163. doi:10.1161/STROKEAHA.110.611731

Cramer S, Simonsen H, Frederiksen J, Rostrup E, Larsson H. Abnormal blood-brain barrier permeability in normal appearing white matter in multiple sclerosis investigated by MRI. NeuroImage Clin. 2014;4:182-189. doi:10.1016/j.nicl.2013.12.001

Wilson G, Bryan J, Cranston K, Kitzes J, Nederbragt L, Teal T. Good enough practices in scientific computing. PLoS Comput Biol. 2017;13:e1005510. doi:10.1371/journal.pcbi.1005510

Wilson G, Aruliah D, Brown C, et al. Best practices for scientific computing. PLoS Biol. 2014;12:e1001745. doi:10.1371/journal.pbio.1001745

van Houdt PJ, Ragunathan S, Berk M, et al. Open Science initiative for perfusion imaging (OSIPI): a community-led, open-source code library for analysis of DCE/DSC-MRI. In: Proceedings of the 31st Annual Meeting of ISMRM. London, UK 2022 p. 2691.

Hanson E, Sandmann C, Malyshev A, Lundervold A, Modersitzki J, Hodneland E. Estimating the discretization dependent accuracy of perfusion in coupled capillary flow measurements. PloS One. 2018;13:e0200521. doi:10.1371/journal.pone.0200521

Blockley NP, Jiang L, Gardener AG, Ludman CN, Francis ST, Gowland PA. Field strength dependence of R1 and R 2* relaxivities of human whole blood to prohance, vasovist, and deoxyhemoglobin. Magn Reson Med. 2008;60:1313-1320. doi:10.1002/mrm.21792

Cheng HLM. Investigation and optimization of parameter accuracy in dynamic contrast-enhanced MRI. J Magn Reson Imaging. 2008;28:736-743. doi:10.1002/jmri.21489

O'Connor JPB, Aboagye EO, Adams JE, et al. Imaging biomarker roadmap for cancer studies. Nat Rev Clin Oncol. 2017;14:169-186. doi:10.1038/nrclinonc.2016.162

OSIPI, Shalom E, Kazerooni AF. TF6.2_DCE-DSC-MRI_Challenges. GitHub. https://github.com/OSIPI/TF6.2_DCE-DSC-MRI_Challenges (accessed June 2022).