This review concluded that automated methods of harm detection are feasible and some can potentially detect patient harm efficiently. Limited analysis makes it difficult to draw conclusions from this review and the authors' conclusions should be interpreted with caution.

To identify, describe and evaluate the effectiveness of automated in-patient harm-detection method.

MEDLINE (from 1950) and CINAHL (from 1982) were searched to August 2008 for published articles in English. Search terms were reported. References of key articles were screened.

In-patient studies that compared an automated harm detection method with an alternative method of harm detection and measured actual harm were eligible for inclusion.

Studies were conducted in adults, children, geriatric patients and all age groups. Where reported, study settings included general medical units, general surgical units, intensive care units, medical subspecialties, neonatal and paediatric intensive care units and obstetric units. Most studies assessed field-defined systems (which relied on computerised detection using pre-existing numeric or coded data stored in medical records) and a small number of studies assessed natural language processing (computerised analysis of free text within a medical record to detect language indicative of harm), some in combination with field-defined systems. Systems were either fully or partially automated. Studies aimed to assess adverse drug events, general adverse events, nosocomial infection or other specific adverse events (such as decubitus ulcers, surgical complications).

The authors did not state how studies were selected for inclusion in the review.

Diagnostic accuracy studies were assessed according to criteria of independent blind comparison with the gold standard, avoidance of partial verification bias and validation of the assessment in a second set of patients.

The authors did not state how many reviewers performed the quality assessment.

Where sufficient data were reported sensitivity, specificity and positive and negative predictive value were calculated.

Two reviewers independently extracted data.

A narrative synthesis was presented based on studies that reported diagnostic accuracy data.

Forty-three studies were included in the review. Fourteen diagnostic cohort studies (number of participants/admission unclear, range 123 to 3,987 where reported) provided accuracy data and were evaluated in more detail. Seven of these applied the gold standard using independent, blind evaluators, eight avoided partial verification bias and only one study validated the results in an independent set of patients.

Sensitivity ranged from 10% to 94%. Specificity ranged from 23% to 98%.

Automated methods of harm detection are feasible and some can potentially detect patient harm efficiently. Effectiveness varied widely and most studies had methodological weaknesses.

The review assessed a broad question and inclusion criteria were defined. The literature search was restricted to published studies in English and so there were risks of language and publication biases. Appropriate steps were taken to minimise bias and errors when extracting data; it was unclear whether such steps were taken when selecting studies and assessing study quality. The quality of the diagnostic accuracy studies was assessed using some relevant criteria and the results of the assessment were reported. The quality of the other studies was not discussed and very limited details of these studies were reported.

A narrative synthesis was appropriate given the differences between studies. The synthesis was very limited and some investigation of the differences between studies to try and identify features associated with better performance of automated detection methods would have been more informative. Results of studies that did not report on the accuracy of the systems were not reported and it was unclear why these were included in the review; if the purpose was simply to describe available systems then it was unclear why studies were required to include a control group.

The limited analysis makes it difficult to draw conclusions from this review and the authors' conclusions should be interpreted with caution.

Practice: The authors did not state any implications for practice.

Research: The authors stated that further work was needed to develop and assess these tools before they can yield accurate estimates of harm that can be reliably interpreted and compared.

Institute for Healthcare Improvement, USA.

This is a critical abstract of a systematic review that meets the criteria for inclusion on DARE. Each critical abstract contains a brief summary of the review methods, results and conclusions followed by a detailed critical assessment on the reliability of the review and the conclusions drawn.