|
The effectiveness of Web-based vs. non-Web-based interventions: a meta-analysis of behavioral change outcomes |
Wantland D J, Portillo C J, Holzemer W L, Slaughter R, McGhee E M |
|
|
CRD summary This review compared the effectiveness of Web-based and non-Web- based interventions to encourage individual changes in behaviour. The authors concluded that there is substantial evidence to support the effectiveness of Web-based interventions at improving behavioural change outcomes. However, this conclusion appears overstated given the substantial differences between the studies, which in some cases were methodologically flawed and failed to report statistically significant effects. Authors' objectives To compare the effectiveness of Web-based and non-Web-based interventions to encourage individual changes in behaviour. Searching MEDLINE, CINAHL, EMBASE, PsycINFO, ERIC and the Cochrane Library were searched from 1996 to 2003; some search terms were reported. In addition, the reference lists of studies and dissertations were checked for further studies. Study selection Study designs of evaluations included in the reviewRandomised controlled trials (RCTs), controlled studies and descriptive studies reporting outcomes at baseline and post-intervention were eligible for inclusion. Specific interventions included in the reviewStudies comparing Web-based and non-Web-based behaviour or educational interventions aimed at encouraging behaviour changes in individuals were eligible for inclusion. The following were excluded from the review: non-Web-based Computer Assisted Instruction interventions; interventions providing website access only; professional practice interventions; telephone-based interventions; interventions incorporating synchronous video communication; class room or non-clinic/non-home-based interventions; provider-focused interventions with no client participation; and remote monitoring interventions. Interventions could be clinic- or home-based. The included interventions were one-off interventions and interventions which lasted up to an average of 27 weeks (range: 3 to 78). Interventions were based on the following conceptual design frameworks (where stated): cognitive restructuring techniques; knowledge change leading to behaviour change; behaviour change; Weinsteins Precaution Adoption Process; transtheoretical (Stages of Change) model; cognitive-behavioural change; cognitive- behavioural therapy; self-efficacy theory; self-help; and Rogers Diffusion of Innovation Theory. Participants included in the reviewThe authors did not specify which types of participants were eligible for inclusion. The average age of the included participants was 41.5 years and, overall, there were roughly equal numbers of men and women. Participants received interventions for the following conditions or reasons: depression, asthma (paediatric), eating disorders, weight control, nutrition, human immunodeficiency virus infection or acquired immunodeficiency syndrome, exercise promotion, cognitive-behavioural therapy, paediatric encopresis, tinnitus, ulcerative colitis, post-traumatic stress disorder and recurrent headache. Outcomes assessed in the reviewStudies assessing behaviour changes and/or self-efficacy health outcomes were eligible for inclusion. The included outcomes and measures varied but details were reported. How were decisions on the relevance of primary studies made?The authors did not state how the papers were selected for the review, or how many reviewers performed the selection. Assessment of study quality Study validity was assessed using a quality rating scale that considered study design, sample selection, specification of illness or condition, reproducibility of the study, and specification of the outcomes and outcome measures. Each study was awarded a score of up to 18 points and only studies scoring at least 12 points were included in the review. The authors did not state how the validity assessment was performed. Data extraction Means and standard deviations were extracted for outcome measures, and effect sizes (with confidence limits) calculated for each study using Hedges' d. Attrition rates were also extracted. Where standard deviations were not reported, but p- values and/or z-scores were, effect sizes were calculated using the Stouffer method. For longitudinal studies with multiple follow-up points, the effect size was calculated using the earliest time operated for the control group and the final time period for the control, then repeated for the intervention in order to achieve one effect size for each study. Means were combined for those studies with multiple Web-based or paper-based interventions. Effect sizes and confidence limits were represented graphically using drop-line charts. Methods of synthesis How were the studies combined?The studies were combined in a narrative. How were differences between studies investigated?Some differences between the studies were evident from the data tables and discussed in the text of the review. Statistical heterogeneity was assessing using the homogeneity statistic (Qw). The studies were grouped according to whether they assessed outcomes at one time point or over a period of time. The authors also reported carrying out sensitivity analyses to ascertain the studies which contributed the greatest amount of heterogeneity. Results of the review Twenty-three studies (n=11,754) including 15 RCTs, a crossover RCT, one case-control study, 2 longitudinal studies, 3 convenience sample studies and a descriptive study, were reported in the review.
Sixteen of the 17 studies assessing effects over time reported improved knowledge and/or improved behavioural outcomes for participants using Web-based interventions; six were statistically significant. Effect sizes varied and were classified as small (+/- 0.01 to 0.19), moderate (+/- 2.0 to 0.47) or moderately large (+/- 0.54 to 0.75). Six studies were reported to have found a statistically significant benefit of Web-based interventions. One study favoured a non-Web-based intervention but was not significant. A high level of heterogeneity was detected between the studies and sensitivity analyses suggested that this was mainly due to 3 studies.
Two of the 5 one-time cross-sectional studies comparing assessment instruments or methods applied to Web-based and non-Web-based interventions, reported negative effect sizes favouring non-Web-based interventions, but none were statistically significant. The remaining 3 studies showed small to moderate positive effect sizes (range: 0.17 to 0.44), one of which was statistically significant. A significant level of statistical heterogeneity was detected. Authors' conclusions Substantial evidence exists to support the effectiveness of Web-based interventions at improving behavioural change outcomes such as increased exercise time, increased knowledge of nutritional status, increased knowledge of asthma treatments, increased participation in health care, slower health decline, improved body shape perception and maintenance of weight loss at 18 months. CRD commentary This review answered a broad research question covering a wide variety of participants, interventions and outcomes. A number of electronic databases were searched, but publication bias may be a problem as the authors do not appear to have made any specific attempts to local unpublished material, although they did report assessing dissertations. There were few details about the review methods so it is difficult to assess whether appropriate steps were taken to prevent reviewer bias and error. The quality of the studies was assessed and only studies meeting a minimum quality score were eligible for inclusion; however, the results were not presented and it is therefore difficult to assess the reliability of the individual study data.
Significant high levels of statistical heterogeneity were detected between studies. The authors also commented that some studies suffered from selection bias and high attrition rates. Overall, given the high level of heterogeneity between studies and the aforementioned methodological problems, the authors' conclusions seem overstated, particularly as only a few of the observed effect sizes were statistically significant. Implications of the review for practice and research Practice: The authors did not state any implications for practice.
Research: The authors stated that further studies to investigate the long-term effects of interventions on the persistence with chosen therapies and cost-effectiveness of web-based therapies are required. Continued software developments also need further evaluation. Bibliographic details Wantland D J, Portillo C J, Holzemer W L, Slaughter R, McGhee E M. The effectiveness of Web-based vs. non-Web-based interventions: a meta-analysis of behavioral change outcomes. Journal of Medical Internet Research 2004; 6(4):e40 Indexing Status Subject indexing assigned by NLM MeSH Adult; Chronic Disease /therapy; Female; Health Behavior; Health Knowledge, Attitudes, Practice; Health Promotion /methods /trends; Humans; Internet /utilization; Male; Outcome Assessment (Health Care); Reproducibility of Results; Research Design; Self Care /methods /statistics & numerical data AccessionNumber 12005004670 Date bibliographic record published 14/12/2006 Date abstract record published 09/08/2008 Record Status This is a critical abstract of a systematic review that meets the criteria for inclusion on DARE. Each critical abstract contains a brief summary of the review methods, results and conclusions followed by a detailed critical assessment on the reliability of the review and the conclusions drawn. |
|
|
|