This review concluded that intra-observer and inter-observer reliability of still-image ultrasound scans for detection of synovitis was high (especially for power Doppler) in studies that involved highly trained observers. Reliability of acquisition of ultrasound should be further assessed. These conclusions should be interpreted with some caution due to the possibility of publication bias.

To determine the reliability of ultrasonographic synovitis detection in rheumatoid arthritis.

PubMed, EMBASE and The Cochrane Library were searched to June 2009. Search terms were reported. Relevant conference proceedings were searched from 2007 to 2008. Reference lists were screened. The review was restricted to published English-language studies.

Studies that reported any information on reliability for detection of synovitis by ultrasound in adults with rheumatoid arthritis were eligible for inclusion.

Included studies used binary, categorical (0 to 3 categories) or quantitative scoring system for synovitis and reported on interobserver and or intraobserver reliability. Joints examined included the metacarpophalangeal (MCP), proximal interphalangeal (PIP), distal interphalangeal (DIP), metatarsophalangeal (MTP), wrist, shoulder, elbow, hip, knee, ankle and foot. Observers were either rheumatologists or radiologists and were experienced in musculoskeletal ultrasonography (except one that included rheumatology fellows with limited ultrasonographic experience). Ultrasonographic mode included B-mode, power Doppler, 3-D power Doppler, and contrast power Doppler alone or in combination. Ultrasonographic data were image acquisition, still images or both. The time to retest for studies that assessed intra-observer reliability ranged from 30 minutes to one year. The number of observers assessed for inter-observer reliability studies ranged from two to 23. Patients had typical active rheumatoid arthritis in 28 joints with Disease Activity Scores that ranged from 4.9 to 5.9.

The authors did not state how`many reviewers assessed studies for inclusion.

Studies were assessed for methodological quality with criteria developed by the authors: reporting of selection criteria; sample size calculation; description of ultrasonographic scanning technique; description of blinding; description of synovitis scoring; and adequate reporting of statistics and results. Studies were assigned a score from 0 to 6 based on the number of items fulfilled.

The authors did not state how many reviewers assessed study quality.

Data were extracted on ultrasonographic mode and the number of ultrasonographers, their level of experience and the type and number of joints involved in reliability testing. Reliability data were extracted for both inter-observer and intra-observer reliability. Method of agreement was expressed as kappa statistics, intraclass correlation coefficient (ICC), coefficient of variation (CV), overall agreement or Kendall's W coefficient. Where multiple statistics were reported, kappa statistics were extracted. If values were reported for each joint or joint group, data were extracted for metacarpophalangeal (MCP) joints, proximal interphalangeal (PIP) joints and wrists.

The authors did not state how many reviewers extracted data.

For studies that reported kappa statistics, linear regression was used to investigate the association of number of joints, quality score and number of observers (inter-observer reliability only) with measures of reliability. Other studies were combined in a narrative synthesis.

Thirty-five studies were included (1,415 patients, range four to 278 patients). All ultrasonographers were blinded to clinical and patient data. None of the studies performed a sample size calculation. The description of scanning technique was variable. Quality scores ranged from 3 to 5 out of 6.

Still Images: Intra-observer reliability was high in B-mode (kappa 0.5 to 1.0; 14,991 joints) and power Doppler (kappa 0.59 to 1.0; 14,934 joints). Interobserver reliability was also high in B-mode (kappa 0.49 to 1.0; 3,138 joints) and power Doppler (kappa 0.66 to 1.0; 3,325 joints).

Image acquisition: Inter-observer reliability was lower than for still images (kappa 0.22 to 0.95). Few studies assessed intra-observer reliability for image acquisition and these reported variable results for B-mode (kappa 0.2 to CV 1.9% to 2.6% suggested excellent reliability; three studies) and power Doppler (ICC 0.95 and CV 4.5% suggested excellent reliability; two studies).

Linear regression analysis found no evidence of an association between any of the factors investigated and inter-observer reliability.

Intra-observer and inter-observer reliability of still-image interpretation was high (especially for power Doppler) in studies that involved highly trained observers. Reliability of acquisition of ultrasound should be further assessed.

The review addressed a clear question and inclusion criteria were defined. The literature search was adequate. The restriction of the review to published English-language studies meant that there was a possibility that relevant studies were missed. Details of the review process were not reported and so it was impossible to determine whether appropriate steps were taken to minimise bias and errors. Study quality was assessed using relevant criteria. Methods to synthesise studies studies were appropriate and the authors conclusions were supported by the results presented. However, the conclusions should be interpreted with some caution due to the possibility of missed studies.

Practice: The authors did not sate any implications for practice.

Research: The authors stated that further studies were required for reliability testing (especially in acquisition image reliability). There was a continuing need for more standardisation of scanning technique and training.

One of the authors was funded through the 2009 Articulum Fellowship.

This is a critical abstract of a systematic review that meets the criteria for inclusion on DARE. Each critical abstract contains a brief summary of the review methods, results and conclusions followed by a detailed critical assessment on the reliability of the review and the conclusions drawn.