This review appeared to conclude that for every million vaccinations there were 60 cases of accidental infection, 40 of generalised vaccinia, 13 of eczema vacciantum, three of post-vaccinial encephalitis and one of vaccinia necrosum. Death rates associated with these complications were reported. The conclusions follow from the evidence presented but the results may not apply to populations vaccinated today.

To calculate age-specific complication and case fatality risks (CRs and CFRs, respectively) after smallpox vaccination in the United States (U.S.).

MEDLINE was searched from 1965 to November 2002; the search terms were stated. In addition, the reference lists of identified studies and reviews were checked. Selected first authors were contacted for additional studies. The review did not include reports published before 1965 or from any other U.S. databases. The searches were restricted to publications in the English language.

Study designs of evaluations included in the review

The studies had to report the number of people vaccinated and their vaccination status (primary or re-vaccination), and provide sufficient data to permit the calculation of CRs and CFRs. The included studies were period studies, prospective cohort studies, or clinic-based retrospective studies.

Specific interventions included in the review

Studies of smallpox vaccination conducted in the U.S. were eligible for inclusion. Most of the included studies obtained data on the number of immunisations from National Bureau of Census Immunization Survey estimates.

Participants included in the review

The inclusion criteria were not specified in terms of participants.

Outcomes assessed in the review

The studies had to report one or more complications of vaccination and use definitions consistent with the criteria of Neff et al. (see Other Publications of Related Interest). The review assessed post-vaccinial encephalitis, vaccinia necrosum, eczema vacciantum, generalised vaccinia and accidental infection. The included studies obtained data on complications from a number of sources, such as the Red Cross vaccinia immune globulin distribution system and surveys of physicians and paediatricians.

How were decisions on the relevance of primary studies made?

Two reviewers independently selected studies for inclusion.

The following potential sources of bias for CRs and CFRs were considered and discussed in the text: study location, the screening of potential vaccines, source reporting vaccine complication and deaths, and source reporting the number of people vaccinated. Validity was assessed, but the authors did not state who performed the validity assessment.

Two reviewers and two other staff members independently extracted the data. The two reviewers resolved any disagreements. Data were extracted on the study characteristics, sources of data and results. The data were grouped by age of vaccinee: younger than 1 year, 1 to 4 years, 5 to 19 years, and older than 19 years. For each complication of primary vaccination, the CR, the mortality risk (MR) and the CFR were calculated for each age group and each study, together with their 95% confidence intervals (CIs). The risks for each complication were also calculated for each study for re-vaccinations.

How were the studies combined?

The studies were combined using a meta-analysis. For primary vaccinations, pooled age-specific risks and 95% CIs were calculated for each complication; the MR and the CFR for each complication were also calculated. Calculations of pooled risks and 95% CIs were based on the binomial distribution. The same pooled risks were calculated for re-vaccinations, but the data in these meta-analyses were not stratified by age.

How were differences between studies investigated?

Differences between the studies were discussed in the text with reference to the methods used to collect the data.

Seven studies were included (approximately 13,206,095 people having primary vaccination and approximately 18,349,359 people having re-vaccination). Of these, five were period studies, one was a prospective cohort study and one was a clinic-based retrospective study.

Primary vaccination.

Post-vaccinial encephalitis (13,206,095 vaccinees): for all age groups combined, the CR per million was 2.9 (95% CI: 2.0, 3.9), the MR per million was 0.8 (95% CI: 0.4, 1.5) and the CFR per hundred was 28.9 (95% CI: 15.4, 45.9). There were 11 deaths among 38 cases of post-vaccinial encephalitis. Infants aged younger than 1 year were at greatest risk for post-vaccinial encephalitis (the risk ratio was 2.80 compared with older vaccinees). One prospective study that surveyed physicians found higher complication rates in infants aged younger than 1 year (176 cases per million versus 6.8 cases per million overall).

Vaccinia necrosum (13,206,095 vaccinees): for all age groups combined, the CR per million was 1.0 (95% CI: 0.5, 1.7), the MR per million was 0.2 (95% CI: 0.02, 0.5) and the CFR per hundred was 15.4 (95% CI: 1.9, 45.4). There were 2 deaths among 13 cases of post-vaccinial encephalitis. Vaccinees aged 20 years or older were at greatest risk for vaccinia necrosum (5.3 cases per million). Most cases occurred in people with previously diagnosed haematological malignancy or impaired immunity.

Eczema vacciantum (13,206,095 vaccinees): for all age groups combined, the CR per million was 12.8 (95% CI: 10.9, 14.9), the MR per million was 0.0 (95% CI: 0.0, 0.3) and the CFR per hundred was 0.0 (95% CI: 0.0, 2.2). There were 0 deaths among 169 cases of eczema vacciantum. There were 3 deaths among 132 cases in non-vaccinees after contact with a vaccinee. There was no difference across age groups.

Generalised vaccinia (13,206,095 vaccinees): for all age groups combined, the CR per million was 39.9 (95% CI: 36.6, 43.5). There were 0 deaths among 527 cases. Infants aged younger than 1 year were at greatest risk for generalised vaccinia (103.3 cases per million; the risk ratio was 2.80 compared with older vaccinees). Within age groups, the highest rates were reported in studies that surveyed physicians.

Accidental infection (13,206,095 vaccinees): for all age groups combined, the CR per million was 64.9 (95% CI: 60.6, 69.4). There were 0 deaths among 857 cases of accidental infection.

Re-vaccination.

The summary risks were lower than for primary vaccination. There were 2 deaths among over 19.5 million vaccinees.

Post-vaccinial encephalitis: the risk was 26 times lower; the CR per million was 0.1 (95% CI: 0.01, 0.4).

Vaccinia necrosum: the risk was 1.5 times lower; the CR per million was 0.7 (95% CI: 0.3, 1.1). There were 2 deaths among 12 cases. The CFR per hundred was 16.7 (95% CI: 2.1, 48.4).

Eczema vacciantum: the risk was 12 times lower; the CR per million was 1.0 (95% CI: 0.6, 1.6).

Generalised vaccinia: the risk was 29 times lower; the CR per million was 1.4 (95% CI: 0.9, 2.0).

Accidental infection: the risk was 3.8 times lower; the CR per million was 16.9 (95% CI: 15.1, 18.9).

The authors' conclusion appears to be that for every million vaccinations there were 60 cases of accidental infection, 40 cases of generalised vaccinia, 13 cases of eczema vacciantum, 3 cases of post-vaccinial encephalitis and 1 case of vaccinia necrosum. They also concluded that the complication rates ascertained using physician surveys were higher than the rates obtained using data from the vaccinia immune globulin programme.

The review question was clear in terms of the intervention and outcomes. Only one database was searched, although the authors did contact some authors for additional studies. Given the focus of the review on U.S. studies, the language restrictions were appropriate. However, this limited search strategy may have missed other relevant studies. Two reviewers independently selected the studies, while two reviewers plus two additional staff members extracted the data, thus reducing the potential for bias and errors. The methods used to assess validity were not described; hence, any efforts made to reduce errors and bias cannot be judged. Some aspects of validity were assessed and summarised in the text, and relevant information on the included studies was tabulated.

The data were combined in a meta-analysis, but heterogeneity was not formally tested. In their discussion, the authors correctly acknowledged that differences between the studies, with respect to screening, diagnosis and the treatment of complications, mean that summary estimates of risk may not be appropriate. The authors also discussed some other limitations of the review, including the use of data from studies conducted 40 years ago in an era of routine smallpox vaccination. The authors pointed out that the current situation differs from the historical one: e.g., there are differences in the diagnosis and treatment of vaccine complications, and a change in the distribution of risk factors for complications among populations who might undergo vaccination. The authors' conclusions follow from the results presented. However, as the authors acknowledged, the results of this review may not apply to populations vaccinated today.

Practice: The authors stated that, at the current time, the risks to an individual of smallpox vaccination are greater than the potential benefits unless there is an outbreak of smallpox.

Research: The authors did not state any implications for further research.

Neff JM, Lane JM, Pert JH, Moore R, Millar JD, Henderson DA. Complications of smallpox vaccination. I. National survey in the United States, 1963. New Engl J Med 1967;276:125-32.

This is a critical abstract of a systematic review that meets the criteria for inclusion on DARE. Each critical abstract contains a brief summary of the review methods, results and conclusions followed by a detailed critical assessment on the reliability of the review and the conclusions drawn.