|
Using an improvement model to reduce adverse drug events in VA facilities |
Weeks W B, Mills P D, Dittus R S, Aron D C, Batalden P B |
|
|
Record Status This is a critical abstract of an economic evaluation that meets the criteria for inclusion on NHS EED. Each abstract contains a brief summary of the methods, the results and conclusions followed by a detailed critical assessment on the reliability of the study and the conclusions drawn. Health technology The intervention was a multi-disciplinary, collaborative, quality improvement (QI) project to reduce medication errors. The project's impact was determined by comparing baseline with end of project data. The comparator was therefore no QI project.
Type of intervention Other: prevention of medication errors.
Economic study type Cost-effectiveness analysis.
Study population The study population comprised patients treated by the VA.
Setting The QI project was offered to all 172 VA centres in the USA. The authors did not report details of the type or care settings of centres that participated.
Dates to which data relate The effectiveness evidence and resources used were collected for a study period during 1999 and 2000. The authors did not state the date of prices used in the study.
Source of effectiveness data The effectiveness data were derived from a single study.
Link between effectiveness and cost data The cost data were derived from published estimates and accounts data, and not from the same patient sample as that used for the effectiveness analysis.
Study sample The authors invited all the study population to participate (172 VA centres). The intervention was designed specifically for the VA. Of these 172 centres, 30 teams expressed interest in the study and 27 completed the initial work. Data were collected from more than 20,000 veterans. Inclusion/exclusion criteria were not appropriate for the sampling framework. The authors did not report whether power calculations were used to assess the adequacy of the enrolled sample for the evaluation. The authors did not report details about the centres that did not participate and limited details about those that did. It is not possible to assess whether the study sample was appropriate for the objectives of the evaluation.
Study design The study design was a before and after study in one group of VA centres and the patients treated in those centres. The QI intervention was implemented over a 7 month time frame. The duration of follow from implementation was 13 months. All of the 27 teams that started the QI project completed the implementation period and 6 months of follow up.
Analysis of effectiveness The evaluation was based on an intention to treat analysis of all the teams that started the QI project. The primary outcome measures used at the 6 month follow up were:
team remained intact;
team maintained gains;
team focus spread to other topics or locations;
number of patients for whom medication errors were averted;
number of patients for whom allergy status was newly documented (2 teams chose to focus on this aspect).
The authors used statistical methods to evaluate whether there were correlations between outcomes and potential confounding factors arising from differences between the teams involvement in the pre-work required before implementation of the project, and barriers to change.
Effectiveness results The primary outcome results at the 6 month follow up were:
team remained intact = 18/27 (66.7%);
team maintained gains = 21/27 (78%);
team focus spread to other topics = 9/27 (33.3%), or locations 15/27 (55.6%);
number of patients for whom medication errors were averted = 3,669;
number of patients for whom allergy status was newly documented = 54,006.
Clinical conclusions The use of the QI intervention appeared to be effective in changing practice and reducing medication errors.
Measure of benefits used in the economic analysis The measure of benefit used in the economic analysis was serious or life threatening adverse drug events prevented. These were estimated as the proportion of medication errors that resulted in serious or life threatening adverse drug events multiplied by the observed number of medication errors averted by the QI project. The proportion of medication errors that resulted in serious or life threatening adverse drug events was estimated from 2 published studies.
Direct costs A discounted cash flow accounting model was developed to analyse net (incremental) costs and return on investment. The costs are shown below.
The actual costs of personnel, travel and contracting to develop and implement the QI intervention.
The direct costs of care for patients with serious or life threatening adverse drug events. These were estimated from 1 published study. The authors do not report how these costs were estimated.
The litigation costs associated with serious or life threatening adverse drug events. Actual settlements and payments for litigation associated with adverse drug events were obtained from the Veteran Administration's claims database.
Returns on investment were estimated by examining two scenarios:
limiting the analysis to improvements and personnel costs that actually occurred between initiation of the project and follow up 13 months later; and
examining a scenario in which the improvements and personnel costs were extended for another 3 years.
The internal rate of return and the net present value were examined in both scenarios. Discount rates of 0%, 5% and 10% were used in the analysis.
The authors did not report the dates of the price data used or whether the cost estimates were adjusted for inflation.
Statistical analysis of costs There were no statistical analyses of costs.
Indirect Costs Indirect costs were not included in the study and were not appropriate to the perspective used.
Currency US dollars ($). No currency conversions were reported.
Sensitivity analysis In addition to the cost analyses above, sensitivity analyses were performed for low and high estimates of medication errors and serious or life threatening adverse events averted.
Estimated benefits used in the economic analysis Between 740 and 2,589 serious or life-threatening adverse events were averted from initiation of the formal project through the 6-month follow-up period.
Cost results The cost of the intervention was reported as $1.7 million. The potential direct care cost savings of serious or life threatening adverse events averted were between $3.47 million (740 serious or life threatening adverse events averted) and $12.13 million (2,589 serious or life threatening adverse events averted). The net costs of the intervention from implementation to the end of the 6 month follow up were therefore between $1.77 million and $10.43 million.
Synthesis of costs and benefits The sensitivity analysis incorporating costs of the project, as well as direct and indirect cost savings, resulted in a range of internal rate returns on the investment.
Scenario 1. For the low estimate and high estimate of occurrence of adverse events, the internal rate of return was 40% and 364%, respectively. For the low estimate of adverse events, the annualised net present value at 0% discount rate was $1,422,577, at 5% $1,014,193 and at 10% $774,665. For the high estimate of adverse events, the annualised net present value was $10,692,770 at 0% discount rate, $9,356,106 at 5%, and $8,253,828 at 10%.
Scenario 2. For the low estimate and high estimate of occurrence of adverse events, the internal rate of return was 46% and 371%, respectively. For the high estimate of adverse events, the annualised net present value was $10,692,770 at 0% discount rate, $9,356,106 at 5% and $8,253,828 at 10%. For the high estimate of adverse events, the annualised net present value was $11,153,474 at 0% discount rate, $10,256,913 at 5%, and $9,489,544 at 10%.
Authors' conclusions The teams successfully completed a collaborative effort to reduce adverse drug events. The investment appeared to be worthwhile from a financial perspective and appeared to increase with time.
CRD COMMENTARY - Selection of comparators The end of project results were compared with baseline data, so that the comparator was usual practice without the QI intervention. The authors did not report what usual practice was before the intervention. The authors indicated that concern about the rate of medication errors was a reason for developing the QI intervention. It is not clear whether other action would have been taken without the QI intervention, and the authors did not report whether there are alternative methods of reducing medication errors. You as a user of this database should decide whether no change in practice is an appropriate response to concern about medication errors in your own setting.
Validity of estimate of measure of effectiveness The study used a before and after design to evaluate the outcomes of the project. The authors used statistical analysis to control for the impact of team characteristics and barriers to change, within the centres that participated. However, the study design and study sample might have introduced bias into the estimates of effectiveness, which were not controlled for by the analysis.
Firstly, the authors noted that only 27 of 172 centres participated in the project and evaluation. This may have introduced selection bias. For example, there may be differences in the rate of medication errors or barriers to change between the centres that did and did not participate. If the participating centres had higher rates of medication errors or fewer barriers to change, then the estimates of effectiveness could be over estimated. The authors reported that barriers to change were correlated to the outcomes measures used.
Secondly, the authors noted that the before and after design of the study meant that it was not possible to control for other factors that might have affected the rate of medication errors, such as organisational changes or changes in usual practice.
Thirdly, the authors noted that staff participating in the project collected the information used to assess outcomes. The single group design meant that it was not possible to mask the research staff to intervention allocation. This may have introduced subjective bias into the analysis, which could over estimate the effectiveness of the QI project if staff were committed to the QI implementation, and/or had a subjective interest in reporting improvements. In addition, the authors did not describe how data on medication errors averted were collected and collated. No definition was given of what constitutes a potential medication error.
Validity of estimate of measure of benefit The measure of benefit was adverse drug events avoided. This seems an appropriate measure. However, the number of adverse drug events avoided was estimated from the number of medication errors avoided; the range used to calculate the potential occurrence of adverse drug events after medication errors was wide (20-70%). The authors did not describe the types of medication errors included in the study. Only life-threatening or serious adverse drug events were included in the analysis. There may be other causes of adverse drug events in addition to medication errors. The proportion of serious or life threatening adverse events due to medication error were derived from 2 published studies. The authors do not report details about the quality or validity of these studies. The measure of benefit used does not provide an evaluation of the impact of reducing medication errors to patients' health or the value of improvements in patients' health and social well being.
Validity of estimate of costs The authors did not report resource use and costs separately. The costs of serious or life threatening adverse events were estimated from 1 published study. The authors did not report details about the quality or validity of that study. The authors used actual expenditure to estimate the costs of the intervention. The authors estimated litigation costs for serious or life threatening adverse events from actual settlements and historical patterns of payment. The authors did not report details of the methods used or whether cost data from different time periods were adjusted to standardise for inflation. The authors discounted the costs of the intervention appropriately. The authors reported sensitivity analyses that indicated the results were robust to large variation in the occurrence of serious or life threatening adverse events. No sensitivity analyses were conducted to test the robustness of the results to the cost estimates used.
Other issues The authors compared the results of the evaluation with those of other evaluations of similar projects and suggested that the results are similar. The authors noted that the VA may be different from other institutions, which may limit the application of the results of the evaluation to other settings. The authors reported limited information about the centres that did participate, which, combined with a lack of demographic information on the patients included in the study, makes it difficult to assess potential differences between settings.
Implications of the study The principles of clinical improvement can be applied successfully in teaching teams how to achieve improvements in care, and these methods are successful within the setting of a single organisation. Using medium- and long-term investment as an outcome measure may provide leaders with an important way to evaluate quality improvement measures. Although the costs are not insignificant, the gains are potentially substantial in terms of improved quality of care, better outcomes and decreased costs.
Source of funding Drs Weeks and Mills were funded by the Veterans Health Administration through the VISN 1 Patient Safety Center of Inquiry. The work of the VA National Quality Scholars Fellowship Program is funded by the Veterans Health Administration; Case Western Reserve University, Dartmouth Medical School, University of Alabama at Birmingham, University of California at San Francisco, University of Iowa, and Vanderbilt University, and a gift from the Pfizer Foundation.
Bibliographic details Weeks W B, Mills P D, Dittus R S, Aron D C, Batalden P B. Using an improvement model to reduce adverse drug events in VA facilities. Joint Commission Journal on Quality Improvement 2001; 27(5): 243-254 Indexing Status Subject indexing assigned by NLM MeSH Cost Savings; Direct Service Costs /statistics & Drug Therapy /standards; Drug-Related Side Effects and Adverse Reactions; Follow-Up Studies; Health Services Research; Hospitals, Veterans /standards; Humans; Inservice Training /organization & Leadership; Medication Errors /economics /prevention & Medication Systems, Hospital /standards; Models, Organizational; Organizational Culture; Organizational Innovation; Patient Care Team /organization & Program Evaluation; Risk Management; Total Quality Management /organization & United States; United States Department of Veterans Affairs; administration; administration; administration; control /statistics & numerical data; numerical data AccessionNumber 22001008113 Date bibliographic record published 30/04/2002 Date abstract record published 30/04/2002 |
|
|
|