Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2020 Mar 26;10(1):5492.
doi: 10.1038/s41598-020-62148-y.

Computer aided detection of tuberculosis on chest radiographs: An evaluation of the CAD4TB v6 system

Keelin Murphy  1 , Shifa Salman Habib  2 , Syed Mohammad Asad Zaidi  2 , Saira Khowaja  3   4 , Aamir Khan  3   4 , Jaime Melendez  5 , Ernst T Scholten  6 , Farhan Amad  2 , Steven Schalekamp  6 , Maurits Verhagen  7 , Rick H H M Philipsen  5 , Annet Meijers  5 , Bram van Ginneken  6
Affiliations

Computer aided detection of tuberculosis on chest radiographs: An evaluation of the CAD4TB v6 system

Keelin Murphy et al. Sci Rep. .

Abstract

There is a growing interest in the automated analysis of chest X-Ray (CXR) as a sensitive and inexpensive means of screening susceptible populations for pulmonary tuberculosis. In this work we evaluate the latest version of CAD4TB, a commercial software platform designed for this purpose. Version 6 of CAD4TB was released in 2018 and is here tested on a fully independent dataset of 5565 CXR images with GeneXpert (Xpert) sputum test results available (854 Xpert positive subjects). A subset of 500 subjects (50% Xpert positive) was reviewed and annotated by 5 expert observers independently to obtain a radiological reference standard. The latest version of CAD4TB is found to outperform all previous versions in terms of area under receiver operating curve (ROC) with respect to both Xpert and radiological reference standards. Improvements with respect to Xpert are most apparent at high sensitivity levels with a specificity of 76% obtained at a fixed 90% sensitivity. When compared with the radiological reference standard, CAD4TB v6 also outperformed previous versions by a considerable margin and achieved 98% specificity at the 90% sensitivity setting. No substantial difference was found between the performance of CAD4TB v6 and any of the various expert observers against the Xpert reference standard. A cost and efficiency analysis on this dataset demonstrates that in a standard clinical situation, operating at 90% sensitivity, users of CAD4TB v6 can process 132 subjects per day at an average cost per screen of $5.95 per subject, while users of version 3 process only 85 subjects per day at a cost of $8.38 per subject. At all tested operating points version 6 is shown to be more efficient and cost effective than any other version.

PubMed Disclaimer

Conflict of interest statement

J.M., R.H.H.M.P., A.M. were in the employment of Thirona (developer of CAD4TB software) at the time of manuscript preparation. B.vG receives royalties and funding from Delft Imaging Systems and Mevis Medical Solutions and stock, royalties and funding from Thirona. The other authors report no conflicts.

Figures

Figure 1
Figure 1
Flowchart describing data collection.
Figure 2
Figure 2
Sample output from CAD4TB v6. (a) The original radiograph, (b) The radiograph with abnormality heatmap overlay. The final composite CAD4TB score for this subject was 91.7 (0 = normal, 100 = most abnormal) and the Xpert test was positive.
Figure 3
Figure 3
Comparing previous and current releases of CAD4TB. (a) Reference = Xpert. N = 5665. (b) Reference = Radiological ‘Consensus’. N = 500. Shaded areas represent the 95% confidence intervals for the curves shown (calculated by bootstrapping).
Figure 4
Figure 4
Each of the 5 observers is compared against a ‘consensus’ reference standard from the remaining 4 observers. In each case the scores of the observer not in the reference standard are thresholded at score values 1 and 2 to obtain two distinct operating points of sensitivity and specificity. The performance of CAD4TB v6 against the same reference standard is also illustrated in each case. Shaded areas representing 95% confidence intervals are calculated by bootstrapping.
Figure 5
Figure 5
Expert observer performance compared with CAD4TB version 6, Reference = Xpert. The observers scored a set of 500 cases (250 positive, 250 negative). The CAD4TB curve is shown for both all 5665 cases (854 positive, 4711 negative), and the 500 annotated cases. (Note that the 500 cases were selected such that CAD4TB v6 Az = 0.885 over the set.) The shaded region of the curves and the error-bars on observer points represent 95% confidence-intervals calculated by bootstrapping.
Figure 6
Figure 6
An illustration of how CAD4TB, used as a pre-screening tool, reduces costs and increases daily throughput. Results for both CAD4TB v6 and CAD4TB v3 are shown at four different sensitivity levels. The inset bar charts illustrate the average cost per screening (CAVG) and the daily throughput of the unit (θ) for each sensitivity level and CAD4TB version. Costing and throughput in the absence of CAD4TB is also illustrated.
Figure 7
Figure 7
Cases where radiograph presentation does not conform with Xpert result, making prediction by radiograph alone difficult for both observers and CAD4TB. The inset images show the CAD4TB heatmaps where blue indicates most normal texture and red indicates most abnormal. (a) An Xpert-negative case marked as TB positive (score 3) by all five observers and by CAD4TB v6 (score = 100). (b) An Xpert-positive case marked with score 1 (no-TB) by 4 of the experts and score 2 by the last one. The CAD4TB score for this case is 18.7 which is not picked up as TB positive until a sensitivity of 99% is reached.
Figure 8
Figure 8
Cases where radiograph interpretation by observers and CAD4TB conforms well with Xpert outcome. The inset images show the CAD4TB heatmaps where blue indicates most normal texture and red indicates most abnormal. (a) An Xpert-positive case marked as TB positive (score 3) by all five observers and by CAD4TB v6 (score = 91.7). (b) An Xpert-negative case marked with score 1 (no-TB) by all 5 experts. The CAD4TB score for this case is 7.1.
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

See all "Cited by" articles

References

    1. World Health Organization, Global Tuberculosis Report, http://apps.who.int/iris/bitstream/handle/10665/274453/9789241565646-eng... (2018).
    1. Horne, D. J. et al. Xpert MTB/RIF and Xpert MTB/RIF Ultra for pulmonary tuberculosis and rifampicin resistance in adults. Cochrane Database of Systematic Reviews, 10.1002/14651858.CD009593.pub4 (2019). - PMC - PubMed
    1. Philipsen, R. H. H. M. et al. Automated chest-radiography as a triage for Xpert testing in resource-constrained settings: a prospective study of diagnostic accuracy and costs. Scientific Reports5, 12215, http://www.ncbi.nlm.nih.gov/pubmed/26212560 http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=PMC4515744 http://www.nature.com/articles/srep12215 (2015). - PMC - PubMed
    1. FIND negotiated product pricing, https://www.finddx.org/find-negotiated-product-pricing/.https://www.finddx.org/find-negotiated-product-pricing/.
    1. World Health Organization Xpert MTB/RIF implementation manual, http://apps.who.int/iris/bitstream/handle/10665/112469/9789241506700_eng... (2014).

Publication types