This review evaluated the effect of publicly reported performance data in a range of different settings. The authors concluded that while there appeared to be some quality improvement activity at the hospital level, elsewhere, the effects on effectiveness, safety and patient-centeredness remain unclear. The authors’ cautious conclusions seem to reflect the evidence presented and are likely reliable.

To assess the effects of the public release of performance data on quality improvement activity, selection of providers, clinical outcomes and unintended consequences.

MEDLINE, EconLit, Web of Science, and Wilson Business Periodicals were searched from 1999 to 2006; the search terms were reported. For articles published before 1999, the reports included in a previous systematic review were considered (see Other Publications of Related Interest). Only articles written in English were included. The bibliographies of relevant published reviews and an unpublished review were checked, and experts in the field were contacted for additional studies.

Study designs of evaluations included in the review

Inclusion criteria were not defined in terms of the study design but it appears that any type of study design was eligible. Selected studies included randomised controlled trials (RCTs), other experimental studies and observational cohort studies.

Specific interventions included in the review

Studies assessing the effects of publicly reported performance data were eligible for the review.

Participants included in the review

Inclusion criteria were not defined in terms of the study participants. The type of participants varied across the studies including, for instance, patients or staff members from academic or community hospitals, medical beneficiaries, adults with private insurance and federal employees.

Outcomes assessed in the review

Studies that evaluated the effects of the public release of performance data on selection of providers, quality improvement activity, clinical outcomes (effectiveness, patient safety and patient-centeredness) and unintended consequences were eligible for the review.

How were decisions on the relevance of primary studies made?

Two authors independently selected the studies, but the authors did not state how any disagreements were resolved.

Two authors independently assessed study quality, with any disagreements resolved by discussion with a third author. The degree of overlap between the participants or reporting system and the characteristics and needs of the privately insured or Medicare population was assigned one to four stars, with four stars indicating complete overlap and one star no overlap. Study design was considered, with four stars assigned to the strongest design and one to the weakest. The weight of the study was assessed on the basis of the strength of the evidence reported, which was adjudicated according to the Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) system.

Two authors independently extracted the data, with any disagreements resolved through discussion.

How were the studies combined?

The data were combined in a narrative.

How were differences between studies investigated?

Differences in study design, study participants and outcomes were discussed in the text and presented in more detail in the tables.

Forty-five articles (the number of participants was unclear) were included in the review.

Overall, health plan–level studies had the highest global ratings, quality improvement activity studies received a low global score, and outcome studies had medium global ratings.

Health plans: Eight studies (two RCTs, four observational cohorts and two experimental studies) examined the effects of public reporting on the selection of health plans. A modest association was found, although the included studies reported conflicting data. A retrospective cohort study suggested that health plans voluntarily reporting performance data outperformed non–publicly reporting plans in technical and patient experience domains. Another retrospective cohort study found that plans with lower quality-of-care scores were more likely than higher-scoring plans to stop publicly reporting their quality data (odds ratio 3.6, 95% confidence interval: 2.1, 7.0).

Hospitals: Nine studies (three observational cohorts, three analysis of time trend and three time series) evaluated the effects of public reporting on hospital selection. Overall, there seemed to be no effect on hospital selection, with mixed findings reported across the studies. Eleven investigations (three case series, one case study, five surveys, one controlled trial and one study of interviews) assessed the effect of publicly releasing performance data on quality improvement activity and, overall, suggested that public reporting stimulates some quality improvement activity. Eleven studies (seven observational studies, two time series, one case study and one controlled trial) that assessed the effect of public performance data on outcomes and six studies (four observational studies, one time series and one case series) that evaluated unintended consequences of public reporting reported conflicting results.

Individual providers: Seven studies (six observational studies and one cross-sectional study) examined the effects of public reporting on individual provider selection and reported contrasting data. One observational cohort study found that risk-adjusted mortality rates for surgeons decreased after the data were released. Six studies focused on whether public reporting caused unintended consequences. Four articles assessed whether publicly reporting performance data negatively affected access to care for more severely ill patients. Three reports found some reluctance among surgeons to operate on high-risk patients after the implementation of public reporting. Two articles evaluated the effect for certain socioeconomic groups. One investigation showed that persons from higher socioeconomic neighbourhoods were more likely to be treated by surgeons with low risk-adjusted mortality rates, whereas persons in lower socioeconomic neighbourhoods were more likely to be treated by surgeons with higher risk-adjusted mortality rates.

Publicly releasing performance data appeared to stimulate quality improvement activity at the hospital level, while the evidence on individual providers and practices and on the effect of public reporting on effectiveness, safety and patient-centeredness was limited.

This review addressed a well-defined question in terms of the interventions and study outcomes, while a broad definition of study participants and design was used. Several databases were searched and some efforts were made to find additional studies by reviewing the reference lists of relevant reviews. Attempts to identify unpublished studies were not made and the potential influence of publication bias was not considered in the report. Only reports written in English were selected, therefore the potential for language bias cannot be excluded. The authors attempted to minimise bias and errors in the review by carrying out critical phases of the review process in duplicate. The authors’ decision not to pool the studies in a meta-analysis was justified given the apparent clinical differences between the studies and the variety of study designs included. The authors’ cautious conclusions seem to reflect the evidence presented and are likely to be reliable.

Practice: The authors did not state any implications for practice.

Research: The authors stated that future studies with a rigorous design should evaluate more existing reporting systems and examine the causal pathways through which public reporting influences quality of care.

RAND Health’s Comprehensive Assessment of Reform Options (COMPARE) Initiative.

Marshall MN, Shekelle PG, Leatherman S, Brook RH. The public release of performance data: what do we expect to gain? A review of the evidence. JAMA 2000;283(14):1866-1874.

Shekell PG, Lim YW, Mattke S, Damberg C. Does public release of performance results improve quality of care: a systematic review. London: The Health Foundation. 2008.

This is a critical abstract of a systematic review that meets the criteria for inclusion on DARE. Each critical abstract contains a brief summary of the review methods, results and conclusions followed by a detailed critical assessment on the reliability of the review and the conclusions drawn.