CHEST RADIOLOGY / ORIGINAL PAPER
Optimising strategies for artificial intelligence-assisted classification of viral pneumonias on CT imaging: a comparative study of selective and default approaches
1
Department of Radiology, ASST Grande Ospedale Metropolitano Niguarda, Milan, Italy
2
Department of Oncology, University of Turin, Italy
3
Department of Medical Physics, ASST Grande Ospedale Metropolitano Niguarda, Milan, Italy
4
Department of Chemical‐Clinical and Microbiological Analyses, ASST Grande Ospedale Metropolitano Niguarda, Milan, Italy
5
Department of Physics, Universy of Milan, Italy
6
Department of Oncology and Hemato-Oncology, Universy of Milan, Italy
Submission date: 2025-04-13
Final revision date: 2025-05-11
Acceptance date: 2025-05-20
Publication date: 2025-08-05
Corresponding author
Francesco Rizzetto
Department of Radiology, ASST Grande Ospedale Metropolitano Niguarda, Piazza dell’Ospedale Maggiore 3, 20162, Milan, Italy
Department of Radiology, ASST Grande Ospedale Metropolitano Niguarda, Piazza dell’Ospedale Maggiore 3, 20162, Milan, Italy
Pol J Radiol, 2025; 90: 384-393
KEYWORDS
TOPICS
ABSTRACT
Purpose:
To evaluate how different artificial intelligence (AI)-powered approaches affect human performance in a demanding chest computed tomography (CT) task, such as distinguishing between viral pneumonias.
Material and methods:
Three radiologists blindly evaluated 220 chest CT scans of viral pneumonia cases (n = 151 COVID-19; n = 69 other viruses), classifying them with a probabilistic scoring system (COVID-19 Reporting and Data System – CO-RADS) in 2 phases: before (S1) and after (S2) receiving AI classifier results. Two S2 scenarios were investigated: a default approach, with AI predictions available for all cases, and a selective approach, with AI limited to equivocal S1 cases (CO-RADS = 3). Inter-reader agreement (Gwet’s AC2) and diagnostic performance were analysed.
Results:
Radiologists demonstrated good-to-excellent agreement across all scenarios (AC2 = 0.77-0.81). Evaluation changes between S1 and S2 occurred in 18% of cases, with 29% of cases initially classified as CO-RADS = 3. In these equivocal cases, AI led to an average correct classification rate of 85%. Conversely, when radiologists were confident in their S1 diagnoses (CO-RADS ≠ 3), classification changes in S2 occurred in 7% of cases, preventing incorrect diagnoses in 45% of patients but resulting in missed correct classifications in 55%. Regarding diagnostic performance, S1 accuracy was 78%, with 15% of CO-RADS = 3 cases. In S2, under the default approach, accuracy increased to 81%, with 16% of CO-RADS = 3 cases, whereas the selective approach achieved 79% accuracy with only 10% of CO-RADS = 3 cases. Only the selective approach significantly reduced the proportion of equivocal cases (p < 0.009).
Conclusions:
A selective AI approach effectively reduces diagnostic uncertainty without introducing unnecessary complexity, emphasising its potential to optimise radiological workflows in challenging diagnostic scenarios.
To evaluate how different artificial intelligence (AI)-powered approaches affect human performance in a demanding chest computed tomography (CT) task, such as distinguishing between viral pneumonias.
Material and methods:
Three radiologists blindly evaluated 220 chest CT scans of viral pneumonia cases (n = 151 COVID-19; n = 69 other viruses), classifying them with a probabilistic scoring system (COVID-19 Reporting and Data System – CO-RADS) in 2 phases: before (S1) and after (S2) receiving AI classifier results. Two S2 scenarios were investigated: a default approach, with AI predictions available for all cases, and a selective approach, with AI limited to equivocal S1 cases (CO-RADS = 3). Inter-reader agreement (Gwet’s AC2) and diagnostic performance were analysed.
Results:
Radiologists demonstrated good-to-excellent agreement across all scenarios (AC2 = 0.77-0.81). Evaluation changes between S1 and S2 occurred in 18% of cases, with 29% of cases initially classified as CO-RADS = 3. In these equivocal cases, AI led to an average correct classification rate of 85%. Conversely, when radiologists were confident in their S1 diagnoses (CO-RADS ≠ 3), classification changes in S2 occurred in 7% of cases, preventing incorrect diagnoses in 45% of patients but resulting in missed correct classifications in 55%. Regarding diagnostic performance, S1 accuracy was 78%, with 15% of CO-RADS = 3 cases. In S2, under the default approach, accuracy increased to 81%, with 16% of CO-RADS = 3 cases, whereas the selective approach achieved 79% accuracy with only 10% of CO-RADS = 3 cases. Only the selective approach significantly reduced the proportion of equivocal cases (p < 0.009).
Conclusions:
A selective AI approach effectively reduces diagnostic uncertainty without introducing unnecessary complexity, emphasising its potential to optimise radiological workflows in challenging diagnostic scenarios.
REFERENCES (38)
1.
Moynihan R, Sanders S, Michaleff ZA, Scott AM, Clark J, To EJ, et al. Impact of COVID-19 pandemic on utilisation of healthcare services: a systematic review. BMJ Open 2021; 11: e045343. DOI: https://doi.org/10.1136/bmjope....
2.
Sagili C. Enhancing diagnostic accuracy with AI: a review of current applications and future directions. Int J Sci Res Comput Sci Eng Inf Technol 2024; 10: 796-805.
3.
Hassan H, Ren Z, Zhao H, Huang S, Li D, Xiang S, et al. Review and classification of AI-enabled COVID-19 CT imaging models based on computer vision tasks. Comput Biol Med 2022; 141: 105123. DOI: https://doi.org/10.1016/j.comp....
4.
Pennisi M, Kavasidis I, Spampinato C, Schinina V, Palazzo S, Proietto F, et al. An explainable AI system for automated COVID-19 assessment and lesion categorization from CT-scans. Artif Intell Med 2021; 118: 102114. DOI: https://doi.org/10.1016/j.artm.... 102114.
5.
Zhang K, Liu X, Shen J, Li Z, Sang Y, Wu X, et al. Clinically applicable AI system for accurate diagnosis, quantitative measurements, and prognosis of COVID-19 pneumonia using computed tomography. Cell 2020; 181: 1423-1433.e11. DOI: https://doi.org/10.1016/j.cell....
6.
Yang W, Sirajuddin A, Zhang X, Liu G, Teng Z, Zhao S, et al. The role of imaging in 2019 novel coronavirus pneumonia (COVID-19). Eur Radiol 2020; 30: 4874-4882.
7.
Rubin GD, Ryerson CJ, Haramati LB, Sverzellati N, Kanne JP, Raoof S, et al. The role of chest imaging in patient management during the COVID-19 pandemic: a multinational consensus statement from the Fleischner Society. Radiology 2020; 296: 172-180.
8.
Tse ZTH, Hovet S, Ren H, Barrett T, Xu S, Turkbey B, et al. AIassisted CT as a clinical and research tool for COVID-19. Front Artif Intell 2021; 4: 590189. DOI: https://doi.org/10.3389/frai.2.... 590189.
9.
Fusco R, Grassi R, Granata V, Setola SV, Grassi F, Cozzi D, et al. Artificial intelligence and covid-19 using chest ct scan and chest x-ray images: machine learning and deep learning approaches for diagnosis and treatment. J Pers Med 2021; 11. DOI: https://doi.org/3390/jpm111009....
10.
Kriza C, Amenta V, Zenié A, Panidis D, Chassaigne H, Urbán P, et al. Artificial intelligence for imaging-based COVID-19 detection: systematic review comparing added value of AI versus human readers. Eur J Radiol 2021; 145: 110028. DOI: https://doi.org/10.1016/j.ejra....
11.
Ni Q, Sun ZY, Qi L, Chen W, Yang Y, Wang L, et al. A deep learning approach to characterize 2019 coronavirus disease (COVID-19) pneumonia in chest CT images. Eur Radiol 2020; 30: 6517-6527.
12.
Meng F, Kottlors J, Shahzad R, Liu H, Fervers P, Jin Y, et al. AI support for accurate and fast radiological diagnosis of COVID-19: an international multicenter, multivendor CT study. Eur Radiol 2022; 33: 4280-4291.
13.
Yang Y, Lure FYM, Miao H, Zhang Z, Jaeger S, Liu J, et al. Using artificial intelligence to assist radiologists in distinguishing COVID-19 from other pulmonary infections. J Xray Sci Technol 2021; 29: 1-17. DOI: https://doi.org/10.3233/XST-20....
14.
van Leeuwen KG, Schalekamp S, Rutten MJCM, van Ginneken B, de Rooij M. Artificial intelligence in radiology: 100 commercially available products and their scientific evidence. Eur Radiol 2021; 31: 3797-3804.
15.
Jussupow E, Spohrer K, Heinzl A. Radiologists’ usage of diagnostic AI systems: the role of diagnostic self-efficacy for sensemaking from confirmation and disconfirmation. Business and Information Systems Engineering 2022; 64: 293-309.
16.
Parekh M, Donuru A, Balasubramanya R, Kapur S. Review of the chest CT differential diagnosis of ground-glass opacities in the COVID era. Radiology 2020; 297: E289-E302. DOI: https://doi.org/10.1148/radiol....
17.
Koo HJ, Choi SH, Sung H, Choe J, Do KH. Radiographics update: radiographic and CT features of viral pneumonia. Radiographics 2020; 40: E8-E15. DOI: https://doi.org/10.1148/rg.202....
18.
Rizzetto F, Perillo N, Artioli D, Travaglini F, Cuccia A, Zannoni S, et al. Correlation between lung ultrasound and chest CT patterns with estimation of pulmonary burden in COVID-19 patients. Eur J Radiol 2021; 138: 109650. DOI: https://doi.org/10.1016/j.ejra....
19.
Garrana SH, Som A, Ndakwah GS, Yeung T, Febbo J, Heeger AP, et al. Comparison of chest CT findings of COVID-19, influenza, and organizing pneumonia: a multireader study. Am J Roentgenol 2021; 217: 1093-1102.
20.
Prokop M, van Everdingen W, van Rees Vellinga T, Quarles van Ufford H, Stöger L, Beenen L, et al. CO-RADS: a categorical CT assessment scheme for patients suspected of having COVID-19 – definition and evaluation. Radiology 2020; 296: E97-E104. DOI: https://doi.org/10.1148/radiol....
21.
Akoglu H. User’s guide to sample size estimation in diagnostic accuracy studies. Turk J Emerg Med 2022; 22: 177. DOI: https://doi.org/10.4103/2452-2....
22.
Bikku T. Multi-layered deep learning perceptron approach for health risk prediction. J Big Data 2020; 7: 50. DOI: https://doi.org/10.1186/s40537....
23.
Zorzi G, Berta L, Rizzetto F, De Mattia C, Felisi MMJ, Carrazza S, et al. Artificial intelligence for differentiating COVID-19 from other viral pneumonias on CT: comparative analysis of different models based on quantitative and radiomic approaches. Eur Radiol Exp 2023; 7: 3. DOI: https://doi.org/10.1186/s41747....
24.
Rizzetto F, Berta L, Zorzi G, Cincotta A, Travaglini F, Artioli D, et al. Diagnostic performance in differentiating COVID-19 from other viral pneumonias on CT Imaging: multi-reader analysis compared with an artificial intelligence-based model. Tomography 2022; 8: 2815-2827.
25.
Wilson EB. Probable Inference, the law of succession, and statistical inference. J Am Stat Assoc 1927; 22: 209-212.
26.
Tran D, Dolgun A, Demirhan H. Weighted inter-rater agreement measures for ordinal outcomes. Commun Stat Simul Comput 2020; 49: 989-1003.
27.
Quarfoot D, Levine RA. How robust are multirater interrater reliability indices to changes in frequency distribution? Am Stat 2016; 70: 373-384.
28.
Vial A, Assink M, Stams GJJM, van der Put C. Safety and risk assessment in child welfare: a reliability study using multiple measures. J Child Fam Stud 2019; 28: 3533-3544.
29.
Berta L, Rizzetto F, De Mattia C, Lizio D, Felisi M, Colombo PEE, et al. Automatic lung segmentation in COVID-19 patients: impact on quantitative computed tomography analysis. Phys Med 2021; 87: 115-122.
30.
Altman D. Practical statistics for medical research. London: Chapman and Hall; 1991.
31.
MacDonald PL, Gardner RC. Type I error rate comparisons of post hoc procedures for I × j chi-square tables. Educational and Psychological Measurement 2000; 60: 735-754.
32.
Wang M, Xia C, Huang L, Xu S, Qin C, Liu J, et al. Deep learning-based triage and analysis of lesion burden for COVID-19: a retrospective study with external validation. Lancet Digit Health 2020; 2: e506-e515. DOI: https://doi.org/10.1016/S2589-....
33.
Zhou M, Yang D, Chen Y, Xu Y, Xu JF, Jie Z, et al. Deep learning for differentiating novel coronavirus pneumonia and influenza pneumonia. Ann Transl Med 2021; 9: 111. DOI: https://doi.org/10.21037/ATM-2....
34.
Wang H, Wang L, Lee EH, Zheng J, Zhang W, Halabi S, et al. Decoding COVID-19 pneumonia: comparison of deep learning and radiomics CT image signatures. Eur J Nucl Med Mol Imaging 2021; 48: 1478-1486.
35.
Bai HX, Wang R, Xiong Z, Hsieh B, Chang K, Halsey K, et al. Artificial intelligence augmentation of radiologist performance in distinguishing COVID-19 from pneumonia of other origin at chest CT. Radiology 2020; 296: E156-165. DOI: https://doi.org/10.1148/radiol.
36.
Mulrenan C, Rhode K, Fischer BM. A literature review on the use of artificial intelligence for the diagnosis of COVID-19 on CT and Chest X-ray. Diagnostics 2022; 12: 869. DOI: https://doi.org/3390/diagnosti....
37.
Fügener A, Grahl J, Gupta A, Ketter W. Cognitive challenges in human–artificial intelligence collaboration: investigating the path toward productive delegation. Information Systems Research 2022; 33: 678-696.
38.
Manirambona E, Okesanya OJ, Olaleke NO, Oso TA, Lucero Prisno DE. Evolution and implications of SARS-CoV-2 variants in the post-pandemic era. Discover Public Health 2024; 21: 16. DOI: https://doi.org/10.1186/s12982....
Share
RELATED ARTICLE
We process personal data collected when visiting the website. The function of obtaining information about users and their behavior is carried out by voluntarily entered information in forms and saving cookies in end devices. Data, including cookies, are used to provide services, improve the user experience and to analyze the traffic in accordance with the Privacy policy. Data are also collected and processed by Google Analytics tool (more).
You can change cookies settings in your browser. Restricted use of cookies in the browser configuration may affect some functionalities of the website.
You can change cookies settings in your browser. Restricted use of cookies in the browser configuration may affect some functionalities of the website.
