|Acute pulmonary embolism: cost-effectiveness analysis of the effect of artificial neural networks on patient care
|Tourassi G D, Floyd C E, Coleman R E
This is a critical abstract of an economic evaluation that meets the criteria for inclusion on NHS EED. Each abstract contains a brief summary of the methods, the results and conclusions followed by a detailed critical assessment on the reliability of the study and the conclusions drawn.
The use of an artificial neural network (ANN) to diagnose and determine treatment strategies in patients with suspected acute pulmonary embolism. The network was trained to predict the probability of pulmonary embolism by using physicians' findings from ventilation-perfusion lung scans and chest radiographs. Responses were categorised into high probability, intermediate probability and low probability at various threshold levels and 3 treatment strategies were compared.
Strategy 1 was to treat (anticoagulation medication) patients in the high category,refer the rest for angiography and treat those with a positive angiogram.
Strategy 2 was to not treat patients in the low category and refer the rest for angiography, treating those with a positive angiography.
Strategy 3 was to treat patients in the high risk group, not treat those in the low risk group, refer the intermediate group for angiography and treat accordingly.
By varying the category threshold levels a total of 10 ANN diagnosis and treatment strategies was analysed, designated ANN 11, 12, 13, 14, 21, 22, 23, 24, 31, and 32.
Economic study type
Patients suspected of having pulmonary embolism (PE) who have inconclusive diagnostic tests and who are therefore currently referred for pulmonary angiography.
Hospital. The study was carried out in the USA.
Dates to which data relate
Clinical data were collected from patients suspected of having PE between 1 April 1992 and 1 July 1993. Input data for the model were taken from articles published between 1988 and 1996. Costs were based on a study published in 1996.
Source of effectiveness data
Effectiveness data were based on a review of the literature.
A decision tree was used to model outcomes and costs. A software package, Treeage, Boston Mass., was used for development and analysis of the model. Possible outcomes were: death following angiography, treated PE, untreated PE, no PE with no treatment and no PE with treatment. Retrospective chart review provided a real cohort of patients whose medical and personal details were tested by the ANN or entered in the model as the control cohort. At a single centre 116 patients suspected of having PE had undergone both lung scanning and pulmonary angiography within 48 hours (this indicated an inconclusive scan) between 1 April 1992 and 1 July 1993 and their charts were reviewed. Eleven of these were excluded from the study because of incomplete charts or scans and, in one case, because the angiography did not produce a conclusive result. This left 104 patients in each hypothetical cohort.
Outcomes assessed in the review
The outcomes assessed in the review were the prevalence of pulmonary embolism in patients with nondiagnostic lung scans, sensitivity and specificity of conventional angiography, mortality and morbidity rates associated with angiography and mortality and morbidity rates due to anticoagulant treatment or lack of treatment in patients suspected of having PE.
Study designs and other criteria for inclusion in the review
Sources searched to identify primary studies
Criteria used to ensure the validity of primary studies
Methods used to judge relevance and validity, and for extracting data
Number of primary studies included
Six studies were reviewed and data from 4 of these were used.
Methods of combining primary studies
Three studies were reviewed for evidence of prevalence of pulmonary embolism in patients with non-diagnostic lung scans. The baseline figure used in the model was approximately that found in one of these studies, although the reason for that choice was not given. Sensitivity and specificity of conventional angiography are said to be standard values. Reference was made to a single published article. Two references were given for mortality and morbidity rates associated with pulmonary angiography. How they were combined was not explained. Morbidity and mortality rates due to anticoagulant treatment was stated to have been "reviewed extensively and summarised", and reference was made to a single published article.
Investigation of differences between primary studies
Results of the review
The prevalence of pulmonary embolism in patients with non-diagnostic lung scans was 18.3% in the study and 20% was the approximation used in the model. The sensitivity of conventional angiography was 98% and specificity was 97%. The mortality rate associated with angiography was 0.5% and the morbidity rate was 1.0%. Mortality and morbidity rates due to anticoagulant treatment or lack of treatment in patients suspected of having PE were as follows:
(1)If a patient has an embolism and is treated, the mortality rate is 8.7% and the morbidity rate 19.2%,
(2) If a patient has no embolism and is treated, the mortality rate is 1.0% and the morbidity rate 11.4%,
(3)If a patient has am embolism and no treatment, the mortality rate is 30% and the morbidity rate 0.0%,
(4)If a patient has no embolism and no treatment both mortality and morbidity rates are 0.0%.
Measure of benefits used in the economic analysis
Lives saved was the underlying measure used but this was not presented in the results. Instead mortality rates at 6 months quoted as a percentage were given for each strategy and for the hypothetical ideal where all cases of PE are correctly identified and treated accordingly. These results were derived from the decision tree model described above which was used to compare many different ANN strategies simply and speedily with actual clinical practice.
Prices and quantities were not analysed separately, although the number of angiograms needed using each strategy was given. No price date was given. Costs were not discounted as the time frame of the study was less than 1 year. Diagnostic costs due to angiography and therapeutic costs were based on a previously published study, (Hull, 1996). An inclusive figure for angiography which includes a 2-day hospital stay was given as $2,500 and an inclusive figure of $6,500 for heparin plus long-term treatment with anticoagulants. These 2 items were the only costs incorporated in the model. Costs common to all strategies such as ventilation-perfusion scanning were excluded. No costs were ascribed to ANN diagnosis since it was stated that no extra personnel were needed and the programme is run on a PC. An average cost per patient under each strategy was produced by combining the costs of each branch of the decision tree and the probabilities of a patient being on that branch. It was not stated whose costs were included in the analysis and the cost boundary was unclear.
Multiple sensitivity analyses were carried out to test uncertainty in the data. A 3-way sensitivity analysis was carried out varying the following parameters simultaneously: cost of treatment, cost of angiography and prevalence of pulmonary embolism. A 2-way sensitivity analysis was performed varying the sensitivity level of the network between 10% and 100% and the prevalence of pulmonary embolism.
Estimated benefits used in the economic analysis
The mortality rate at 6 months was:
using strategy 11, 2.8;
using strategy 12, 3.0;
using strategy 13, 3.0;
using strategy 14, 3.0;
using strategy 21, 2.1;
using strategy 22, 2.2;
using strategy 23, 2.5;
using strategy 24, 2.7 ;
using strategy 31, 3.2;
using strategy 32, 3.2.
Using angiography for every patient the mortality rate at 6 months was 2.3 and, had perfect diagnosis been achieved, the mortality rate would have been 1.7.
The average cost per patient using each strategy was as follows: strategy 11 was $5,851, strategy 12 was $5,170, strategy 13 was $4,809, strategy 14 was $4,681, strategy 21 was $2,934, strategy 22 was $1,899, strategy 23 was $1,349, strategy 24 was $1,153, strategy 31 was $1,377, and strategy 32 was $1,275. Using angiography for every patient the average cost was $3,923 and had no diagnostic methods been used, but all true positive cases been treated and no true negative cases been treated (the hypothetical "ideal" strategy), the average cost per patient would have been $1,300.
Synthesis of costs and benefits
In fact strategies 21 and 22 had mortality rates and costs lower than the comparator and were therefore dominant strategies. However in this study the cost-effectiveness of each strategy was compared to a do-nothing strategy where no costs are incurred and no treatment is given. An incremental analysis was not performed. The cost per life saved using each strategy was given: strategy 11 was $181,992, strategy 12 was $169,749, strategy 13 was $159,664, strategy 14 was $154,872, strategy 21 was $75,437, strategy 22 was $50,130, strategy 23 was $38,391, strategy 24 was $34,613, strategy 31 was $48,394 and strategy 32 was $46,220. Using angiography for every patient the cost per life saved was $107,180 and in the hypothetical "ideal" scenario the cost per life saved would have been $30,233.
If a high survival rate is the overriding factor the network should be used using strategy 2 and set with a low decision threshold to achieve a high sensitivity level (21 or 22). A prospective clinical trial is needed to objectively test the cost-effectiveness of ANNs but this study shows that they may have a role in improving the treatment of clinically challenging patients. Sensitivity analysis showed that the ANN must achieve a sensitivity of at least 80% for a mortality rate comparable with or better than conventional angiography at the typical 10-50% prevalence of pulmonary embolism.
CRD COMMENTARY - Selection of comparators
The use of current clinical practice (all angiograms) as the comparator is clear, however this study also compares results with the hypothetical ideal of perfect diagnosis followed by treatment and with a "no treatment" reference point for the synthesis of costs and benefits. This has obscured the finding that strategy 2, in two of its threshold settings, was dominant over the comparator and that all other strategies had worse clinical outcomes than the comparator.
Validity of estimate of measure of effectiveness
The authors have not shown that their review of the literature was sufficiently comprehensive to make their results valid. Sensitivity analysis showed many areas of uncertainty and the authors themselves call for further study using a prospective trial. Deaths avoided is a simple and convenient measure, but the authors include a morbidity rate in the study which they do not use in the economic evaluation. They do not indicate the severity of disability or illness that is implied but the lowest morbidity rates are not found using the same strategies as those which produce the lowest mortality and are associated with the lowest average costs. It would therefore be a worthwhile exercise to combine mortality and morbidity using a quality of life measure in a cost-utility analysis.
Validity of estimate of costs
It is not acceptable to place no price on the use of an ANN. Capital cost and training the system and user should have been considered as well as running costs. In general there was too little detail about costs. By accepting cost figures from another study without giving many more details the authors have obscured the dates and conditions to which the costs refer as well as the cost boundary. It is impossible to generalise from these costs or to assess their validity.
While the authors concluded that a prospective clinical study is needed this should include a prospective collection of cost data alongside it so that a valid cost-effectiveness study is produced.
Tourassi G D, Floyd C E, Coleman R E. Acute pulmonary embolism: cost-effectiveness analysis of the effect of artificial neural networks on patient care. Radiology 1998; 206(1): 81-88
Other publications of related interest
Hull R D et al. Cost-effectiveness of pulmonary embolism diagnosis. Archives of Internal Medicine 1996;156:68-72.
Subject indexing assigned by NLM
Acute Disease; Angiography /economics; Cost-Benefit Analysis; Costs and Cost Analysis; Diagnosis, Computer-Assisted /economics /methods; Humans; Lung /radiography /radionuclide imaging; Neural Networks (Computer); Pulmonary Artery /radiography; Pulmonary Embolism /economics /mortality /radiography /radionuclide imaging; Referral and Consultation; Sensitivity and Specificity
Date bibliographic record published
Date abstract record published