The authors concluded that the effects of quality improvement collaboratives on quality of care were uncertain and may at best have only a moderate effect on outcomes. Despite some limitations in the search, the review was generally well-conducted and these cautious conclusions are likely to be reliable.

To assess the effectiveness of quality improvement collaboratives in improving quality of care.

The updated review (see Other Publications of Related Interest) also assessed the determinants of successful interventions. This does not form part of the following abstract.

MEDLINE, CINAHL, EMBASE, PsycINFO and The Cochrane Library were searched from 1995 to June 2006 (MEDLINE search updated March 2008). Search terms were reported. Reference lists of eligible studies were handsearched. The search was limited to articles written in English.

Studies of quality improvement collaboratives in a health care setting were eligible for inclusion. A quality improvement collaborative was defined as fulfilling the following five criteria: a specific topic was addressed; major variation between current and best practice; clinical and quality improvement experts provided ideas and support for improvement; there was cooperation between interdisciplinary teams on multiple sites; a model for improvement sets targets and measures change; and collaborative process involves a series of structured activities within a set time-frame.

Interventions in the controlled studies were based on either Vermont Oxford Network (provides a co-ordinated data-driven programme of research, education, quality improvement and feedback on care of very low birth weight infants) or Breakthrough Series approach (six- to 15-month programme facilitating sharing of learning between different organisations in a focused topic area). Some studies that used the Breakthrough Series approach included elements of a chronic care model. The Vermont Oxford Network approach included an ongoing infrastructure of support beyond the time frame of the quality improvement collaboration. Interventions in the uncontrolled studies were often embedded in ongoing quality initiatives.

Eligible studies reported the effect of intervention on care processes or outcomes of care. Outcomes in the controlled studies in the review included a wide range of measures (such as patient education, routine surveillance, drug use, quality of life, laboratory results, symptoms, clinical outcomes, functional status, rates of organ donation and cost). Outcomes in these studies were assessed by chart/dataset review or telephone interview/survey. There were no specific criteria with respect to study design, but the review included randomised controlled trials (RCTs), controlled and uncontrolled before-and-after studies, and case reports.

Two reviewers independently assessed the eligibility of each study.

Validity assessment was based on the checklist of the Cochrane Effective Practice and Organisation of Care Review Group. It addressed the following factors: design, randomisation method, comparability of control sites, protection against bias (such as baseline equality, blinding and avoidance of contamination), reliability of outcome measures and management of losses to follow-up.

Two reviewers independently assessed validity. Any discrepancies were resolved by consensus.

Findings in controlled studies that were statistically significant (p<0.05) were reported in a table. Binary data were reported as the proportions in each group that experienced the outcome. Continuous data were reported as means in each group. Statistical data from uncontrolled studies were not reported in the review.

Two reviewers independently extracted data. Any discrepancies were resolved by consensus.

Findings of controlled studies were combined in a narrative synthesis grouped by study design and by the type of intervention. Heterogeneity between studies was discussed in the text.

The review included nine controlled studies (12 articles) and an unspecified number of uncontrolled studies (60 articles). The controlled studies comprised two RCTs (79 intervention group providers and 78 control providers), one interrupted time series (one provider) and six controlled before-and-after studies (115 intervention group providers, 208 control providers).

In the update, an additional controlled before-and-after study was included (95 intervention group providers and 125 control providers). A further seven uncontrolled studies were retrieved, but it was not possible to carry out an effectiveness evaluation.

Most of the nine controlled studies had major methodological flaws. These included possible differences in baseline measurement (six studies), limited information about control sites, no clear indication that blinding was used and possible contamination. This applied also to the additional study in the update, although baseline measurement was comparable and follow-up was 100%. The uncontrolled studies did not contain adequate information about data collection or analysis and measures were not objective and commonly consisted of self-report measures by participating providers. Almost all of the uncontrolled studies were methodologically weak and probably biased towards positive findings.

Among the controlled studies there were marked differences between the interventions. Four studies contained elements of the chronic care programme in the intervention; two others included ongoing support beyond the time-frame of the quality improvement collaboration. Therefore, it was not possible to determine to the extent to which any effects were related to the quality improvement collaboration.

Breakthrough Series (three studies and one additional study in the update):

Two studies reported positive findings associated with the intervention. One before-and-after study set in nursing homes reported a statistically significant reduction in the proportion of patients with pain (7.2% versus 11.2%, p<0.05). Another study (an interrupted time series of infant mortality in the community) reported a statistically significant increase in mean length of time between deaths (55 days versus 114 days, p<0.05) after a five-year quality programme during which a one-year Breakthrough Series project was conducted. One before-and-after study set in HIV clinics reported no statistically significant effect for any outcome; this included measures of screening, prophylaxis and access to care.

In the updated review, a study that measured conversion rate for organ donation (eligible donors who became actual donors) showed a pre-post period change of 8% (95% confidence interval 2% to 13%) in favour of the collaborative intervention. However, the authors acknowledged that causality was unclear.

Breakthrough Series plus chronic care model (four studies):

A RCT among children with asthma reported no statistically significant effect associated with the intervention on any key processes or intermediate outcomes. Three before-and-after studies reported mixed effects, with statistically significant improvements for some measures. One of these studies focused on management of diabetes in primary care and found statistically significant benefits associated with the intervention in measures of end-organ surveillance, glycaemic control and control of hypertension (p<0.05). A before-and-after study of asthma in children showed statistically significant improvements (p<0.05) in measures of self-management (such as peak flow monitoring) and education (such as use of metered inhalers), but no significant improvement in satisfaction with care or in functional measures. A before-and-after study of patients with chronic heart failure found statistically significant improvements (p<0.05) in measures of counselling, education and communication (for example, related to medications, diet, exercise and goal setting), but performance rates remained low (<50%) for most educational indicators and there was no significant improvement in outcomes such as diagnostic indicators or heart failure symptoms. The process measures that improved the most were those with the lowest initial performance rates.

Vermont Oxford Network (two studies):

Two studies of pre-term infants reported mixed effects. One cluster RCT found a statistically significant increase in rates of oxygen supplementation and a statistically significant decrease in infection rates in the intervention group (both p<0.05), but there was no evidence of a significant improvement in death rates or pneumothorax. A before-and-after study reported a statistically significant improvement associated with the intervention in several measures; these included decreased infection rates in six neonatal units (12.3% versus 16.5%, p<0.05) and in the rate of supplemental oxygen use in four units (34% versus 38.7%). There was no significant improvement in other measures, such as rates of death and pneumothorax.

Findings of uncontrolled studies:

Most of the accounts of uncontrolled studies (88%) reported improvements in patient care and organisational performance, in some cases of 30% to 80%. However, given methodological limitations of these studies, no conclusions on effectiveness could be drawn. Details of these studies were not reported in the review but could be obtained from the authors on request.

One controlled before-and-after study of a quality improvement collaborative using the Vermont Oxford approach in 19 neonatal units reported a significantly reduced cost in one of two cost-effectiveness measures used. Median treatment cost per infant fell from $57,606 at baseline to $45,874 after the 36-month intervention.

The effects of quality improvement collaboratives were uncertain and may at best have only a moderate effect on outcomes.

The review objectives and inclusion criteria were clear. Relevant sources were searched. The restriction to English-language studies may have meant that some studies were missed. It appeared that no specific attempts were made to locate unpublished studies. Two reviewers assessed study selection. Steps were taken to minimise reviewer error in data extraction and quality assessment. Relevant criteria were used to assess study quality. The decision not to report findings of the uncontrolled studies in detail appeared justified. The authors highlighted statistically significant findings in the higher quality studies, although no measures of variance were reported. Heterogeneity between studies was well addressed in the text. The authors noted that their definition of quality improvement collaboration may have introduced bias if included studies differed systematically from similar interventions that involved other features and they discussed other potential sources of bias. Despite some limitations in the search, the review was generally well-conducted and the cautious conclusions are likely to be reliable.

Practice: The authors did not state any implications for practice.

Research: The authors stated that more evidence was needed on which components of quality improvement collaboratives were effective and cost-effective, and the effect of contextual factors and site characteristics on outcomes.

The update review was funded by The Health Foundation.

Hulscher M, Schouten L, Grol R. Collaboratives. The Health Foundation, London. Quest for Quality and Improved Performance. 2009.

This is a critical abstract of a systematic review that meets the criteria for inclusion on DARE. Each critical abstract contains a brief summary of the review methods, results and conclusions followed by a detailed critical assessment on the reliability of the review and the conclusions drawn.