DeepImageTranslator V2: Analysis of Multimodal Medical Images using Semantic Segmentation Maps Generated through Deep Learning

En Zhou Ye, En Hui Ye, Maxime Bouthillier, Run Zhou Ye


Introduction: Analysis of multimodal medical images often requires the selection of one or many anatomical regions of interest (ROIs) for extraction of useful statistics. This task can prove laborious when a manual approach is used. We have previously developed a user-friendly software tool for image-to-image translation using deep learning. Therefore, we present herein an update to the DeepImageTranslator V2 software with the addition of a tool for multimodal medical image segmentation analysis (hereby referred to as the MMMISA). Methods: The MMMISA was implemented using the Tkinter library; backend computations were implemented using the Pydicom, Numpy, and OpenCV libraries. We tested our software using 4188 slices from whole-body axial 2-deoxy-2-[18F]-fluoroglucose-position emission tomography/ computed tomography scans ([¹⁸F]-FDG-PET/CT) of 10 patients from the American College of Radiology Imaging Network-Head and Neck Squamous Cell Carcinoma (ACRIN-HNSCC) database. Using the deep learning software DeepImageTranslator, a model was trained with 36 randomly selected CT slices and manually labelled semantic segmentation maps. Utilizing the trained model, all the CT scans of the 10 HNSCC patients were segmented with high accuracy. Segmentation maps generated using the deep convolutional network were then used to measure organ specific [¹⁸F]-FDG uptake. We also compared measurements performed using the MMMISA and those made with manually selected ROIs. Results: The MMMISA is a tool that allows user to select ROIs based on deep learning-generated segmentation maps and to compute accurate statistics for these ROIs based on coregistered multimodal images. We found that organ-specific [¹⁸F]-FDG uptake measured using multiple manually selected ROIs is concordant with whole-tissue measurements made with segmentation maps using the MMMISA tool.


Doi: 10.28991/HIJ-2022-03-03-07

Full Text: PDF


Multimodal Medical Imaging; Semantic Segmentation; Regions of Interest; Deep Learning; Software; PET/CT Imaging.


Lensen, K. J. D. F., Van Sijl, A. M., Voskuyl, A. E., Van Der Laken, C. J., Heymans, M. W., Comans, E. F. I., Nurmohamed, M. T., Smulders, Y. M., & Boellaard, R. (2017). Variability in quantitative analysis of atherosclerotic plaque inflammation using 18F-FDG PET/CT. PLoS ONE, 12(8), 181847. doi:10.1371/journal.pone.0181847.

Cao, Y., Gathaiya, N., Han, Q., Kemp, B. J., & Jensen, M. D. (2019). Subcutaneous adipose tissue free fatty acid uptake measured using positron emission tomography and adipose biopsies in humans. American Journal of Physiology. Endocrinology and Metabolism, 317(2), E194–E199. doi:10.1152/ajpendo.00030.2019.

Noll, C., Montastier, É., Amrani, M., Kunach, M., Frisch, F., Fortin, M., Bouffard, L., Dubreuil, S., Phoenix, S., Cunnane, S. C., Guérin, B., Turcotte, E. E., Laville, M., & Carpentier, A. C. (2020). Seven-day overfeeding enhances adipose tissue dietary fatty acid storage and decreases myocardial and skeletal muscle dietary fatty acid partitioning in healthy subjects. American Journal of Physiology - Endocrinology and Metabolism, 318(2), E286–E296. doi:10.1152/ajpendo.00474.2019.

Hojjati, M., Badve, C., Garg, V., Tatsuoka, C., Rogers, L., Sloan, A., Faulhaber, P., Ros, P. R., & Wolansky, L. J. (2018). Role of FDG-PET/MRI, FDG-PET/CT, and Dynamic Susceptibility Contrast Perfusion MRI in Differentiating Radiation Necrosis from Tumor Recurrence in Glioblastomas. Journal of Neuroimaging, 28(1), 118–125. doi:10.1111/jon.12460.

Montastier, É., Zhou Ye, R., Noll, C., Bouffard, L., Fortin, M., Frisch, F., Phoenix, S., Guérin, B., Turcotte, É. E., Lewis, G. F., & Carpentier, A. C. (2021). Increased postprandial nonesterified fatty acid efflux from adipose tissue in prediabetes is offset by enhanced dietary fatty acid adipose trapping. American Journal of Physiology - Endocrinology and Metabolism, 320(6), E1093–E1106. doi:10.1152/AJPENDO.00619.2020.

Labbé, S. M., Grenier-Larouche, T., Noll, C., Phoenix, S., Guérin, B., Turcotte, E. E., & Carpentier, A. C. (2012). Increased myocardial uptake of dietary fatty acids linked to cardiac dysfunction in glucose-intolerant humans. Diabetes, 61(11), 2701–2710. doi:10.2337/db11-1805.

Ye, R. Z., Montastier, É., Noll, C., Frisch, F., Fortin, M., Bouffard, L., ... & Carpentier, A. C. (2022). Total Postprandial Hepatic Nonesterified and Dietary Fatty Acid Uptake Is Increased and Insufficiently Curbed by Adipose Tissue Fatty Acid Trapping in Prediabetes With Overweight. Diabetes, 71(9), 1891-1901. doi:10.2337/db21-1097.

Sourceforge Website (2022). DeepImageTranslator: a free, user-friendly tool for image translation using deep-learning and its applications in CT image analysis. Available online: (accessed on February 2022).

Kinahan, P., Muzi, M., Bialecki, B., Herman, B., & Coombs, L. (2019). Data from the ACRIN 6668 Trial NSCLC-FDG-PET. Cancer Imaging Arch, 10.

Lowe, V. J., Duan, F., Subramaniam, R. M., Sicks, J. R. D., Romanoff, J., Bartel, T., Jian, Q., Nussenbaum, B., Richmon, J., Arnold, C. D., Cognetti, D., & Stack, B. C. (2019). Multicenter trial of [18F]fluorodeoxyglucose positron emission tomography/computed tomography staging of head and neck cancer and negative predictive value and surgical impact in the n0 neck: Results from acrin 6685. Journal of Clinical Oncology, 37(20), 1704–1712. doi:10.1200/JCO.18.01182.

Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., & Prior, F. (2013). The cancer imaging archive (TCIA): Maintaining and operating a public information repository. Journal of Digital Imaging, 26(6), 1045–1057. doi:10.1007/s10278-013-9622-7.

Loening, A. M., & Gambhir, S. S. (2003). AMIDE: A Free Software Tool for Multimodality Medical Image Analysis. Molecular Imaging, 2(3), 131–137. doi:10.1162/153535003322556877.

Ye, R. Z., Noll, C., Richard, G., Lepage, M., Turcotte, É. E., & Carpentier, A. C. (2022). DeepImageTranslator: A free, user-friendly graphical interface for image translation using deep-learning and its applications in 3D CT image analysis. SLAS Technology, 27(1), 76–84. doi:10.1016/j.slast.2021.10.014.

Lessard, R., Tremblay, N. M., Plourde, M. É., & Guillot, M. (2020). An open-source software for monitoring intrafraction motion during external beam radiation therapy based on superimposition of contours of projected ROIs on cine-MV images. Journal of Applied Clinical Medical Physics, 21(8), 173–182. doi:10.1002/acm2.12940.

Yushkevich, P. A., Pashchinskiy, A., Oguz, I., Mohan, S., Schmitt, J. E., Stein, J. M., Zukić, D., Vicory, J., McCormick, M., Yushkevich, N., Schwartz, N., Gao, Y., & Gerig, G. (2019). User-Guided Segmentation of Multi-modality Medical Imaging Datasets with ITK-SNAP. Neuroinformatics, 17(1), 83–102. doi:10.1007/s12021-018-9385-x.

Bücking, T. M., Hill, E. R., Robertson, J. L., Maneas, E., Plumb, A. A., & Nikitichev, D. I. (2017). From medical imaging data to 3D printed anatomical models. PLoS ONE, 12(5), 178540. doi:10.1371/journal.pone.0178540.

Debus, C., Floca, R., Ingrisch, M., Kompan, I., Maier-Hein, K., Abdollahi, A., & Nolden, M. (2019). MITK-ModelFit: A generic open-source framework for model fits and their exploration in medical imaging - design, implementation and application on the example of DCE-MRI. BMC Bioinformatics, 20(1), 31. doi:10.1186/s12859-018-2588-1.

Razeghi, O., Solís-Lemus, J. A., Lee, A. W. C., Karim, R., Corrado, C., Roney, C. H., de Vecchi, A., & Niederer, S. A. (2020). CemrgApp: An interactive medical imaging application with image processing, computer vision, and machine learning toolkits for cardiovascular research. SoftwareX, 12, 100570. doi:10.1016/j.softx.2020.100570.

Croteau, E., Lavallée, É., Labbe, S. M., Hubert, L., Pifferi, F., Rousseau, J. A., Cunnane, S. C., Carpentier, A. C., Lecomte, R., & Bénard, F. (2010). Image-derived input function in dynamic human PET/CT: Methodology and validation with 11C-acetate and 18F- fluorothioheptadecanoic acid in muscle and 18F-fluorodeoxyglucose in brain. European Journal of Nuclear Medicine and Molecular Imaging, 37(8), 1539–1550. doi:10.1007/s00259-010-1443-z.

Lee, S. Y., Jee, W. H., Yoo, I. R., Jung, J. Y., Im, S. A., Chung, Y. G., & Kang, J. H. (2019). Comparison of 3T diffusion-weighted MRI and 18F-FDG PET/CT in musculoskeletal tumours: Quantitative analysis of apparent diffusion coefficients and standardized uptake values. British Journal of Radiology, 92(1102), 20181051. doi:10.1259/bjr.20181051.

Richard, G., Noll, C., Archambault, M., Lebel, R., Tremblay, L., Ait‐Mohand, S., Guérin, B., Blondin, D. P., Carpentier, A. C., & Lepage, M. (2020). Contribution of perfusion to the 11 C-acetate signal in brown adipose tissue assessed by DCE‐MRI and 68Ga-DOTA PET in a rat model. Magnetic Resonance in Medicine, 85(3), 1625–1642. Portico. doi:10.1002/mrm.28535.

Carreau, A. M., Noll, C., Blondin, D. P., Frisch, F., Nadeau, M., Pelletier, M., Phoenix, S., Cunnane, S. C., Guérin, B., Turcotte, E. E., Lebel, S., Biertho, L., Tchernof, A., & Carpentier, A. C. (2020). Bariatric surgery rapidly decreases cardiac dietary fatty acid partitioning and hepatic insulin resistance through increased intra-abdominal adipose tissue storage and reduced spillover in type 2 diabetes. Diabetes, 69(4), 567–577. doi:10.2337/db19-0773.

Ye, R. Z., Richard, G., Gévry, N., Tchernof, A., & Carpentier, A. C. (2022). Fat Cell Size: Measurement Methods, Pathophysiological Origins, and Relationships with Metabolic Dysregulations. Endocrine Reviews, 43(1), 35–60. doi:10.1210/endrev/bnab018.

Carpentier, A.C. (2021). 100thanniversary of the discovery of insulin perspective: Insulin and adipose tissue fatty acid metabolism. American Journal of Physiology - Endocrinology and Metabolism (AJPENDO), 320(4), 653–670. doi:10.1152/AJPENDO.00620.2020.

Full Text: PDF

DOI: 10.28991/HIJ-2022-03-03-07


  • There are currently no refbacks.

Copyright (c) 2022 En Zhou Ye, En Hui Ye, Maxime Bouthillier, Run Zhou Ye