Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2020 Sep;296(3):E166-E172.
doi: 10.1148/radiol.2020201874. Epub 2020 May 8.

COVID-19 on Chest Radiographs: A Multireader Evaluation of an Artificial Intelligence System

Keelin Murphy  1 , Henk Smits  1 , Arnoud J G Knoops  1 , Michael B J M Korst  1 , Tijs Samson  1 , Ernst T Scholten  1 , Steven Schalekamp  1 , Cornelia M Schaefer-Prokop  1 , Rick H H M Philipsen  1 , Annet Meijers  1 , Jaime Melendez  1 , Bram van Ginneken  1 , Matthieu Rutten  1
Affiliations

COVID-19 on Chest Radiographs: A Multireader Evaluation of an Artificial Intelligence System

Keelin Murphy et al. Radiology. 2020 Sep.

Abstract

Background Chest radiography may play an important role in triage for coronavirus disease 2019 (COVID-19), particularly in low-resource settings. Purpose To evaluate the performance of an artificial intelligence (AI) system for detection of COVID-19 pneumonia on chest radiographs. Materials and Methods An AI system (CAD4COVID-XRay) was trained on 24 678 chest radiographs, including 1540 used only for validation while training. The test set consisted of a set of continuously acquired chest radiographs (n = 454) obtained in patients suspected of having COVID-19 pneumonia between March 4 and April 6, 2020, at one center (223 patients with positive reverse transcription polymerase chain reaction [RT-PCR] results, 231 with negative RT-PCR results). Radiographs were independently analyzed by six readers and by the AI system. Diagnostic performance was analyzed with the receiver operating characteristic curve. Results For the test set, the mean age of patients was 67 years ± 14.4 (standard deviation) (56% male). With RT-PCR test results as the reference standard, the AI system correctly classified chest radiographs as COVID-19 pneumonia with an area under the receiver operating characteristic curve of 0.81. The system significantly outperformed each reader (P < .001 using the McNemar test) at their highest possible sensitivities. At their lowest sensitivities, only one reader significantly outperformed the AI system (P = .04). Conclusion The performance of an artificial intelligence system in the detection of coronavirus disease 2019 on chest radiographs was comparable with that of six independent readers. © RSNA, 2020.

PubMed Disclaimer

Figures

Top: Images in a 74-year-old man with positive reverse transcription polymerase chain reaction (RT-PCR) test results for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) viral infection. (a) Frontal chest radiograph. (b) Artificial intelligence (AI) system heat map overlaid on a shows pneumonia-related features. The AI system score for this subject is 99.8. Bottom: Images in a 30-year-old man with negative RT-PCR test results for SARS-CoV-2 viral infection. (c) Frontal chest radiograph. (d) AI system heat map overlaid on c. The AI system score for this subject is 0.2.
Figure 1a:
Top: Images in a 74-year-old man with positive reverse transcription polymerase chain reaction (RT-PCR) test results for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) viral infection. (a) Frontal chest radiograph. (b) Artificial intelligence (AI) system heat map overlaid on a shows pneumonia-related features. The AI system score for this subject is 99.8. Bottom: Images in a 30-year-old man with negative RT-PCR test results for SARS-CoV-2 viral infection. (c) Frontal chest radiograph. (d) AI system heat map overlaid on c. The AI system score for this subject is 0.2.
Top: Images in a 74-year-old man with positive reverse transcription polymerase chain reaction (RT-PCR) test results for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) viral infection. (a) Frontal chest radiograph. (b) Artificial intelligence (AI) system heat map overlaid on a shows pneumonia-related features. The AI system score for this subject is 99.8. Bottom: Images in a 30-year-old man with negative RT-PCR test results for SARS-CoV-2 viral infection. (c) Frontal chest radiograph. (d) AI system heat map overlaid on c. The AI system score for this subject is 0.2.
Figure 1b:
Top: Images in a 74-year-old man with positive reverse transcription polymerase chain reaction (RT-PCR) test results for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) viral infection. (a) Frontal chest radiograph. (b) Artificial intelligence (AI) system heat map overlaid on a shows pneumonia-related features. The AI system score for this subject is 99.8. Bottom: Images in a 30-year-old man with negative RT-PCR test results for SARS-CoV-2 viral infection. (c) Frontal chest radiograph. (d) AI system heat map overlaid on c. The AI system score for this subject is 0.2.
Top: Images in a 74-year-old man with positive reverse transcription polymerase chain reaction (RT-PCR) test results for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) viral infection. (a) Frontal chest radiograph. (b) Artificial intelligence (AI) system heat map overlaid on a shows pneumonia-related features. The AI system score for this subject is 99.8. Bottom: Images in a 30-year-old man with negative RT-PCR test results for SARS-CoV-2 viral infection. (c) Frontal chest radiograph. (d) AI system heat map overlaid on c. The AI system score for this subject is 0.2.
Figure 1c:
Top: Images in a 74-year-old man with positive reverse transcription polymerase chain reaction (RT-PCR) test results for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) viral infection. (a) Frontal chest radiograph. (b) Artificial intelligence (AI) system heat map overlaid on a shows pneumonia-related features. The AI system score for this subject is 99.8. Bottom: Images in a 30-year-old man with negative RT-PCR test results for SARS-CoV-2 viral infection. (c) Frontal chest radiograph. (d) AI system heat map overlaid on c. The AI system score for this subject is 0.2.
Top: Images in a 74-year-old man with positive reverse transcription polymerase chain reaction (RT-PCR) test results for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) viral infection. (a) Frontal chest radiograph. (b) Artificial intelligence (AI) system heat map overlaid on a shows pneumonia-related features. The AI system score for this subject is 99.8. Bottom: Images in a 30-year-old man with negative RT-PCR test results for SARS-CoV-2 viral infection. (c) Frontal chest radiograph. (d) AI system heat map overlaid on c. The AI system score for this subject is 0.2.
Figure 1d:
Top: Images in a 74-year-old man with positive reverse transcription polymerase chain reaction (RT-PCR) test results for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) viral infection. (a) Frontal chest radiograph. (b) Artificial intelligence (AI) system heat map overlaid on a shows pneumonia-related features. The AI system score for this subject is 99.8. Bottom: Images in a 30-year-old man with negative RT-PCR test results for SARS-CoV-2 viral infection. (c) Frontal chest radiograph. (d) AI system heat map overlaid on c. The AI system score for this subject is 0.2.
Receiver operating characteristic (ROC) curve for the artificial intelligence (AI) system and points for each reader (point locations are specified in the figure legend). Reference standard is reverse transcription polymerase chain reaction (RT-PCR) test result. The 95% confidence intervals are shown as a shaded area for the ROC curve, and crosshairs indicate each reader point. The AI system operating points discussed in the text are shown at sensitivities of 60%, 75%, and 85%. The test data set has 454 patients (223 with positive RT-PCR results and 231 with negative RT-PCR results). AUC = area under the ROC curve.
Figure 2:
Receiver operating characteristic (ROC) curve for the artificial intelligence (AI) system and points for each reader (point locations are specified in the figure legend). Reference standard is reverse transcription polymerase chain reaction (RT-PCR) test result. The 95% confidence intervals are shown as a shaded area for the ROC curve, and crosshairs indicate each reader point. The AI system operating points discussed in the text are shown at sensitivities of 60%, 75%, and 85%. The test data set has 454 patients (223 with positive RT-PCR results and 231 with negative RT-PCR results). AUC = area under the ROC curve.
Receiver operating characteristic (ROC) curves for the artificial intelligence system and each reader individually. Reference standard in each case is the consensus reading of the remaining five readers. The 95% confidence intervals are shown as a shaded area for the ROC curve. AUC = area under ROC curve.
Figure 3:
Receiver operating characteristic (ROC) curves for the artificial intelligence system and each reader individually. Reference standard in each case is the consensus reading of the remaining five readers. The 95% confidence intervals are shown as a shaded area for the ROC curve. AUC = area under ROC curve.
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

See all "Cited by" articles

References

    1. Yang W, Sirajuddin A, Zhang X, et al. The role of imaging in 2019 novel coronavirus pneumonia (COVID-19). Eur Radiol Published online April 15, 2020. Accessed April 2020. - PMC - PubMed
    1. Rubin GD, Ryerson CJ, Haramati LB, et al. The role of chest imaging in patient management during the COVID-19 pandemic: a multinational consensus statement from the Fleischner Society. Radiology Published online April 7, 2020. Accessed April 2020. - PMC - PubMed
    1. Cheng MP, Papenburg J, Desjardins M, et al. Diagnostic testing for severe acute respiratory syndrome-related coronavirus-2: a narrative review. Ann Intern Med Published online April 13, 2020. Accessed April 2020. - PMC - PubMed
    1. Jacobi A, Chung M, Bernheim A, Eber C. Portable chest X-ray in coronavirus disease-19 (COVID-19): a pictorial review. Clin Imaging 2020;64:35–4210.1016/j.clinimag.2020.04.001. - DOI - PMC - PubMed
    1. COVID-19 situation update for the WHO African Region. World Health Organization. https://apps.who.int/iris/bitstream/handle/10665/331763/SITREP_COVID-19_.... Published April 15, 2020. Accessed April 2020.

LinkOut - more resources