The review compared the accuracy of automated diagnostic instruments with human diagnosis for cutaneous melanoma. No generalisation could be made for all instruments because of differences in technology and applications. Of the three instruments considered in detail, two showed significantly inferior specificity to specialists. Despite methodological weaknesses in the review, the authors' interpretation of the data presented was appropriately cautious.

To compare the accuracy of fully automated diagnostic instruments for cutaneous melanoma with that of human (clinician) diagnosis in a real-world clinical setting.

PubMed, the Cochrane library, EMBASE, Clinical Evidence and CINAHL were searched from 1987 to 2007. Search terms were reported. Update searches (January 1st 2007 to June 1st 2007) were performed in PubMed.

Studies that compared the performance of automated diagnostic instruments with human diagnosis, on an independent test set, were eligible for inclusion. Studies using cross-validation, without using an independent test set, were excluded. Included studies were required to report sensitivity or specificity for the diagnosis of melanoma, or data comparing lesion management based on instrument and human diagnoses. Studies that did not meet these criteria, but were of instruments that had satisfied these requirements elsewhere, were also examined.

The majority of studies were of pigmented skin lesions; the remainder were of melanocytic skin lesions. The proportion of benign lesions ranged from 4 to 86% (where reported). Included studies assessed a range of automated instruments, but the main results of the review focused on DB-MIPS (Dell’Eva-Burroni Dermoscopy Melanoma Image Processing Software from Italy) image analysis software, telespectrometry, or electrical impedance.

Studies were assessed for inclusion by one reviewer, with uncertainties resolved through consultation with an expert in the field and consensus.

The authors did not state that they assessed study validity.

Data were extracted on the sensitivity and specificity of automated instruments and clinicians for the diagnosis of cutaneous melanoma; details of the experience of diagnosing clinicians and 95% confidence intervals (CIs) were also extracted, where reported.

The authors did not state how data were extracted for the review.

Studies were combined in a narrative synthesis.

Nine studies, assessing a total of 4213 lesions, were included in the review.

Only three instruments (four studies) were compared with a human diagnosis in the clinical field with a meaningful sample size that could allow some generalization with the wider clinical arena.

DB-MIPS (two studies): One study showed no significant difference between the instrument and human diagnosis, for either sensitivity or specificity. The other study showed similar sensitivities for instrument and human, with the human (trained dermatologist) showing higher specificity (human diagnosis 99%, 95% CI 98 to 100; instrument diagnosis 74%, 95% CI 69 to 79) when a clinician with minimal training was used, specificities were similar.

Telespectrometry (one study): This study showed similar sensitivities (no significant difference) for the telespectrometry system (80%) and clinical specialists (surgical oncologists with more than five years experience; 91%). The specificity of telespectrometry was found to be significantly lower (49%) than that of the specialists (74%).

Electrical impedance (one study): There was no significant difference between clinician diagnosis and the instrument for sensitivity. However, specificity was significantly poorer for the instrument (58%) than for the clinician (81%) specificity. The instrument could not discriminate melanoma on the head or neck due to different impedance properties of the skin at these sites.

Data from studies of other instruments with smaller sample sizes and studies comparing clinical management as the endpoint rather than diagnosis were reported separately.

A generalization of all automated instruments for the diagnosis of melanoma was not appropriate because of differences in technology and applications. Only three instruments have had their diagnostic accuracy compared with a human diagnosis in the clinical field with a meaningful sample size that could allow some generalization with the wider clinical arena. Two of these instruments showed a significantly inferior specificity for the diagnosis of melanoma compared with specialists.

The review had a clearly stated research objective, but the description of inclusion criteria was limited and the reference standard used to confirm diagnoses was not described. A range of bibliographic databases were searched, decreasing the likelihood that relevant studies were missed. It was not clear whether any restrictions by language or publication status were applied to the searches.

No assessment of the methodological quality of included studies was reported and reporting of the review process was limited; the potential for error and/or bias at study or review level could not be assessed. Some details of the included studies were reported, but the reference standard used to confirm diagnoses was not described. The use of a narrative synthesis was appropriate, given the heterogeneous nature of the included studies.

Despite some methodological weaknesses in the review, the authors' interpretation of the data presented was appropriately cautious.

Practice: The authors did not state recommendations for practice.

Research: Further studies, comparing automated instruments to human diagnosis, the clinical arena are required. This should allow assessment of the non-melanoma set (specificity), which
may vary dramatically as the nature of benign lesions varies from study to study. In addition, assessment of an instrument's ability to detect clinically difficult melanoma, in comparison with humans, is needed. Future studies should include lesions not preselected for excision with follow-up.

Supported in part by NSW Melanoma Network, Australia.

This is a critical abstract of a systematic review that meets the criteria for inclusion on DARE. Each critical abstract contains a brief summary of the review methods, results and conclusions followed by a detailed critical assessment on the reliability of the review and the conclusions drawn.