|Performance and cost evaluation of one commercial and six in-house conventional and real-time reverse transcription-PCR assays for detection of severe acute respiratory syndrome coronavirus
|Mahony J B, Petrich A, Louie L, Song X Y, Chong S, Smieja M, Chernesky M, Loeb M, Richardson S
This is a critical abstract of an economic evaluation that meets the criteria for inclusion on NHS EED. Each abstract contains a brief summary of the methods, the results and conclusions followed by a detailed critical assessment on the reliability of the study and the conclusions drawn.
Seven reverse transcription-polymerase chain reaction (RT-PCR) assays, including six in-house assays and one commercial assay, were compared for the detection of severe acute respiratory syndrome coronavirus (SARS-CoV) RNA in clinical specimens. The probes and primers for each assay were as follows:
assay 1, BNIoutS2, BNIoutAs, BNIinS, BNIAs;
assay 2, BNIoutS2, BNIoutAs;
assay 3, Cor-p-F2, Cor-p-R1;
assay 4, BNIoutS2, BNIoutAs;
assay 5, APNF, APNR;
assay 6, APNF, APNR, Probe; and
assay 7, proprietary.
The RT-PCR assays targeted different genomic regions. There were three conventional assays (one nested and two non-nested) run on a conventional heat block and four real-time assays performed in a LightCycler (LC; Roche Diagnostics). All in-house assays were optimised for assay parameters such as MgCl2, primer, and probe concentrations. The commercial assay was the RealArt HPA CoV RT-PCR assay (Artus), which was run in the LightCycler.
Economic study type
The population comprised hospitalised patients with a probable or suspected diagnosis of SARS.
The setting was unclear, but it was likely to have been tertiary care. The economic study was carried out in Toronto, Canada.
Dates to which data relate
The effectiveness evidence was collected between March and April of 2003. The dates for the resources and prices used were not reported.
Source of effectiveness data
The effectiveness data were derived from a single study.
Link between effectiveness and cost data
The costing was undertaken on the same patient sample as that used in the effectiveness study.
The sample consisted of specimens collected from hospitalised patients with a probable or suspected diagnosis of SARS. Power calculations, sample size and the method of sample selection were not reported. A total of 68 specimens, comprising 17 respiratory tract specimens, 29 urine samples and 22 stool samples, were analysed.
This was a diagnostic accuracy study that was conducted in a single centre.
Analysis of effectiveness
The primary health outcomes used in the analysis were the sensitivity and specificity of each test. Sensitivity was calculated as the percentage positive, for each test, of clinical specimens from suspected SARS patients that were determined to be positive in at least two of the seven assays. Specificity was calculated as the percentage negative, for each test, of clinical specimens that tested positive in none or one of the seven assays.
The Cochrane Q test was used to compare the sensitivities and specificities of the seven assays. Pairwise comparisons were then made using the McNemar test. To compare the relative sensitivities of the various assays more precisely, probit regression analysis was used to estimate the sample dilution at which each test detected 50% of samples with five replicate aliquots of log10 serial dilutions of SARS-CoV RNA. A p-value of 0.05 (two-tailed) was considered statistically significant.
Sixty-three specimens had the same results in all seven assays; 46 were negative and 17 were positive, with only 5 discordant specimens. By the criteria required there were, in total, 18 positives and 50 negatives.
The sensitivities of the seven assays were similar, with sensitivities ranging from 83.3 to 100%. However, differences in the sensitivities were not significant (e.g. assay 2 (18 of 18) versus assay 6 (17 of 18), p=0.5; assay 2 (18 of 18) versus assay 6 (15 of 18), p=0.25).
The specificities of the assays ranged from 94.0 to 100% and were not significantly different (e.g. assay 2 (47 of 50) versus assay 1 (50 of 50), p=0.25; assay 4 (48 of 50) versus assay 1 (50 of 50), p=0.5).
None of the assays was both 100% sensitive and specific. Two of the five discordant specimens were positive in two tests, while the other three were positive in a single test, suggesting that most of the discordant results were false-positive results.
The evaluation of the seven PCR assays revealed that, despite their different formats, the seven different assays performed similarly. The differences in sensitivities and specificities between the seven assays were not significant. This lack of significant differences in sensitivity was surprising, as the most sensitive assays have analytical sensitivities 16 to 166 times higher than the other assays (dilution for 50% detection by probit analysis). Since the definition of the true positives was arbitrary, although it was justified, assay performances could change if a different reference standard was used.
Measure of benefits used in the economic analysis
No summary measure of benefit was used in the economic evaluation. In effect, the authors carried out a cost-consequences analysis.
The costs included for each PCR assay were the reagent component cost and the salary cost, using "actual purchase prices". The cost of RNA extraction was excluded since it was the same for all assays. The costs were based on a run size of 48 samples for a conventional heat block assay, or a run size of 32 on the LC, with three controls run in each assay format. The quantities and the costs were analysed separately and were derived from actual data, but were not reported in detail. The source of the cost data was not reported. Discounting was, appropriately, not performed. The price year was not reported.
Statistical analysis of costs
No statistical tests were reported.
No indirect costs were reported.
Canadian dollars (Can$). The conversion rate was Can$1.00 = US$0.72.
No sensitivity analysis was reported.
Estimated benefits used in the economic analysis
See the 'Effectiveness Results' section.
The total costs per test for the in-house assays were as follows:
assay 1, Can$6.25;
assay 2 and 3, Can$5.46;
assay 4, $8.37;
assay 5, Can$9.30; and
assay 6, Can$9.81.
This cost was compared with a cost of Can$40.37 per test for the commercial test (assay 7). Although it was excluded from all tests, the cost of RNA extraction was reported (Can$4.18).
Synthesis of costs and benefits
The costs and benefits were not combined.
All seven assays were highly specific, and none of them amplified RNA from animal coronaviruses (CoVs) or human respiratory viruses, including CoV OC43, human metapneumovirus, influenza A virus, respiratory syncytial virus, parainfluenza virus types 1, 2, and 3, or DNA from adenovirus.
Five assays showed similar analytical sensitivity with the same RNA end point dilutions, whereas two assays had detection end points at one or two higher dilutions. When the dilutions were tested in replicates of five and the results were analysed by probit regression analysis, the assays demonstrated significant differences in sensitivity.
The evaluation of seven different polymerase chain reaction (PCR) assays for severe acute respiratory syndrome (SARS)-CoV revealed that, despite their different formats, the seven different assays performed similarly.
No conclusion was drawn from the economic analysis.
CRD COMMENTARY - Selection of comparators
The justification for the comparators was the absence of commercially available tests and of any published comparative data on the sensitivity and specificity of in-house RT-PCR assays for the detection of SARS-CoV. You should judge whether these assays are relevant in your own setting, or whether other comparators could also have been relevant, including PCR assay tests.
Validity of estimate of measure of effectiveness
The measure of effectiveness was derived from a single-centre analysis, which might be prone to bias and might limit the validity of the study. Also, there may be concerns about the small sample size and the authors acknowledged that this should be taken into account when considering the negative results. The authors stated that although they made an arbitrary decision to define true-positive specimens as those that were positive in at least two tests, the decision was nevertheless guided by the fact that, had they defined a positive specimen as one that was positive in a single test, then the specificity of all assays would have been 100%, which would clearly have been unrealistic.
Validity of estimate of measure of benefit
No summary measure of benefit was derived and, in effect, the authors carried out a cost-consequences analysis. The reader is thus referred to the comments in the 'Validity of estimate of measure of effectiveness' field (above).
Validity of estimate of costs
The authors did not explicitly report the perspective of the study. There was too little detail on the cost estimation. The costs and the quantities were not reported in sufficient detail to enable the analysis to be easily extrapolated to other settings. The sources of the cost data were not reported. All these factors could affect the robustness of the cost results. A statistical analysis of the costs was not reported. Discounting was not necessary since all the costs were incurred during less than two years. The price year was not reported, which will present difficulties in terms of any future reflation exercise.
The authors did not compare their findings with those from other studies. They also did not directly address the issue of the generalisability of the results to other settings although they stated that, because of small sample size of clinical specimens, additional evaluations with larger numbers of clinical specimens might be required to determine whether there are significant differences in the performance of various assays for the detection of SARS-CoV in clinical specimens.
The authors also reported that the lack of significant differences in sensitivity with clinical specimens might be due to the fact that clinical specimens contain amplification inhibitors that co-purify with RNA and adversely affect different assays, or, alternatively, the similar sensitivity that they observed for the different assays might have been due to the high viral load of SARS-CoV in the clinical specimens used in the study. One might expect a reduced sensitivity for some assays when specimens with lower viral loads are tested. This possibility could be examined in future studies by correlating viral loads with PCR results for various assays.
Implications of the study
Since seroconversion might take as long as 28 days post infection in some SARS patients, the laboratory diagnosis of SARS will continue to rely heavily on the detection of viral RNA by PCR. Given the unknown specificity of available SARS PCR tests in current use, and the obvious consequences of reporting a SARS false-positive result, laboratories would be wise to confirm PCR-positive specimens by using a second assay that targets a different part of the genome. The use of a second confirmatory PCR with a different amplification target will provide laboratories with some assurance that specimens giving positive PCR results are true positives. With this in mind, the authors developed an LC assay (assay 6) whose performance has recently been validated in a multi-centre evaluation.
Although the in-house tests were cheaper than the commercially available test, the additional costs for in-house quality controls that were included in the commercial test would bring the prices closer together. For laboratories setting up SARS testing for the first time without pedigreed specimens and controls, the commercial test might offer a quick start-up.
Mahony J B, Petrich A, Louie L, Song X Y, Chong S, Smieja M, Chernesky M, Loeb M, Richardson S. Performance and cost evaluation of one commercial and six in-house conventional and real-time reverse transcription-PCR assays for detection of severe acute respiratory syndrome coronavirus. Journal of Clinical Microbiology 2004; 42(4): 1471-1476
Subject indexing assigned by NLM
Costs and Cost Analysis; Humans; RNA, Viral /analysis; Reagent Kits, Diagnostic /economics; Reverse Transcriptase Polymerase Chain Reaction /economics /methods; SARS Virus /genetics /isolation & Sensitivity and Specificity; Severe Acute Respiratory Syndrome /virology; purification
Date bibliographic record published
Date abstract record published