The authors state that internet-based learning shows large positive effects compared with no intervention. Effectiveness is similar to that of traditional learning methods. This is a well-conducted review, but the reliability of the authors' conclusions is unclear given the high levels of variation between studies and the possibility of publication bias.

To assess the effectiveness of internet-based learning on learning outcomes in the health professions.

MEDLINE, EMBASE, CINAHL, Scopus, ERIC, TimeLit, Web of Science, Dissertation Abstracts and the University of Toronto Research and Development Resource Base were searched from 1990 to 17 January 2008 for articles in any language. Search terms were reported. Reference lists of included studies, previous reviews and authors' files were handsearched. Meeting abstracts were excluded.

Controlled or repeated measures studies of internet-based learning in health professionals at any stage of training or practice were eligible for inclusion. In order to be eligible for inclusion, controlled studies should have a no intervention or non-internet intervention comparison group. Outcomes eligible for inclusion were learner reaction or satisfaction, learning, and practice behaviours or effects on patients.

Included studies were of internet-based instruction covering diagnostic and therapeutic practice and other areas detailed in the review. Duration of included courses ranged from less than or equal to one day or more than or equal to one month. Some included studies combined internet instruction with face-to-face discussion. Comparison interventions included face-to-face courses, paper modules, satellite-mediated video conferences, slide-tape self study modules and standardized patients. Both classroom and practice settings were included. Participants in included studies were medical students and other health professionals either in training or in practice as physicians, nurses, dentists and pharmacists. A small number were of other health professionals. Outcomes reported in the included studies were: knowledge as assessed by multiple-choice tests or other objectively scored methods or self-report; skills such as communication with patients, critical appraisal, medication dosing cardiopulmonary resuscitation or lumbar puncture measured using objective instructor or standardised patient observations; and behaviour/patient effects such as osteoporosis screening rates, cognitive behaviour therapy implementation and other perceived changes in practice. Included study designs were parallel or crossover randomised or non-randomised controlled studies and within subjects designs.

Two reviewers independently selected studies for inclusion, with disagreements resolved by consensus.

Validity was assessed using an adaptation of the Newcastle-Ottawa Scale for grading the quality of cohort studies. This rated studies according to the representativeness of the intervention group, the selection of the control group, comparability of cohorts, blinding of outcome assessment and completeness of follow up (maximum potential score of 6). Validity assessment was carried out independently and in duplicate with disagreements resolved by consensus.

For each outcome, the means and standard deviations (SD) were extracted to convert to standardized mean differences using Hedges g effect sizes (ES). When insufficient data were available, reported tests of significance (e.g. p values) were used to estimate the ES. If neither p values nor any measure of variance was reported the authors used the average SD from all the other included studies. Data extracted was entered into a standardized data extraction form independently and in duplicate for all variables where reviewer judgement was required. Disagreements were resolved by consensus. Where more than one comparison was used the data were extracted for the intervention most closely resembling the internet intervention. Authors were contacted to supply missing data.

Pooled effects sizes with 95% confidence intervals (CI) were calculated using a random-effects model. The studies were weighted, but the method of weighting was not stated. Effect sizes were calculated separately for learner satisfaction, knowledge, skills and behaviours/effects on patients. Heterogeneity was assessed using the Ι² statistic. Potential sources of heterogeneity were investigated by conducting subgroup analyses. Sensitivity analyses were carried out excluding low-quality studies, studies of blended interventions, studies with effect sizes derived from inexact methods and one study with multiple interventions.

Two hundred and one articles were included for review (n = more than 26,000); 77 post-test-only multiple group designs (n=9,881), 55 pre-test-post-test multiple group designs (n= 6,689; n not available for one study) and 69 repeated measures designs (n=9,562). Sample sizes ranged from 3 to 1,437 participants.

The mean quality score was 2.5 (SD 1.3) out of a potential 6 for no-intervention controlled studies and 3.5 (SD 1.4) for non-internet comparison studies. Loss to follow up was high (more than 25%) for many studies.

Comparison with no intervention (130 studies n=19,234).

Internet-based learning was associated with significant gains in knowledge (117 studies reporting on 126 interventions: pooled effect size = 1.00, 95% CI: 0.90, 1.10, p<0.001), skills (16 studies, pooled effect size 0.85, 95% CI:0.49, 1.20, p<0.001) and behaviours/effects on patient care (19 studies reporting 32 interventions, pooled effect size = 0.82, 95% CI: 0.63, 1.02, p<0.001). There was evidence of significant heterogeneity for all of these outcomes (Ι² = 93.6%, 92.7% and 79.1%). Subgroup analyses revealed that low-quality studies demonstrated larger effect sizes for knowledge (mean score 1.07, 95% CI: 0.96, 1.18 versus mean score = 0.71 95% CI: 0.51, 0.92 p<0.003). Practice exercises were associated with a significant gain in skills (pooled effect size = 1.01, 95% CI: 0.60, 0.43 versus pooled effect size = 0.21, 95% CI: 0.04, 0.38 p<0.001) but with significantly worse behaviour and effect on patient outcomes (pooled effect size = 0.44, 95% CI: 0.33, 0.55 versus pooled effect size = 2.09, 95% CI: 1.38, 2.79, p<0.001). The use of tutorials, online peer discussion and longer duration courses were associated with significantly better outcomes for behaviours/effect on patients (no statistical data provided). High interactivity, access to ongoing course materials and online discussion were not associated with an increase in knowledge or skills.

Comparison with non-internet-based learning.

Internet-based learning was associated with a small but statistically significant increase in knowledge compared to non-internet-based learning methods (63 studies, pooled effect size = 0.12, 95% CI: 0.003, 0.24, p<0.045). There was evidence of significant heterogeneity (Ι² = 88.1%). When blended interventions were removed in a sensitivity analysis the effect became non-significant. Internet learning did not significantly increase satisfaction, skills, or behaviours and effects on patient care when compared to non-internet-based learning. Subgroup analyses revealed that online courses using dicussion yielded significantly higher effects for knowledge (p=0.002), skills (p≤0.04) and behaviours/effect on patients (p=0.02). High levels of interactivity and the use of practice exercises increased the effect size for skills (p≤0.04) but not for knowledge. Single instance interventions showed greater effects in improving skills (p=0.02) and behaviour and effect on patients (p=0.006) when compared to ongoing courses. Short courses and single instance courses were associated with higher levels of learner satisfaction.

Internet-based learning shows large positive effects compared with no intervention. Effectiveness is similar to that of traditional learning methods.

The review addressed a clear question with broad but well-defined inclusion criteria. Several relevant sources were searched and appropriate steps were taken to minimise language bias. Steps do not appear to have been taken to identify unpublished data and publication bias was not assessed. Therefore, publication bias cannot be ruled out. Appropriate steps were taken to minimise reviewer error and bias in all aspects of the review process. A validity assessment was carried out and the results of this were used to inform analyses. Given the presence of significant statistical and clinical heterogeneity between included studies a narrative synthesis may have been more appropriate. This was a generally well-conducted review, but the reliability of the authors' conclusions is unclear given the high levels of heterogeneity between studies and the possibility of publication bias.

Practice: The authors state that a wide range of internet-based education may be used effectively in health-related education.

Research: The authors state that further research is needed comparing different internet-based interventions and comparing internet and non-internet-based learning using outcomes sensitive to the intervention and to change. Further research is also needed to identify variables associated with positive outcomes in internet learning, such as learning context, learning objectives and implementation of internet-based learning programmes.

Intramural funds

Mayo Foundation Education Innovation Award.

This is a critical abstract of a systematic review that meets the criteria for inclusion on DARE. Each critical abstract contains a brief summary of the review methods, results and conclusions followed by a detailed critical assessment on the reliability of the review and the conclusions drawn.