Statistics: Tools for Systematic Review and Meta-Analysis

A. Resource

B. Guidelines

  • Cooper & Hedges, 1994
  • Hedges & Olkin, 1985
  • Lipsey & Wilson, 2001
  • Borenstein, Hedges, Higgins, & Rothstein, 2008: Comprehensive Meta-Analysis Version 2.2.048

C. Review Process

  • Identification of studies
    • Name of the reviewer
    • Date of the review
    • Article: Author, date of publication, title, journal, issue number, pages, and credentials
  • General Information
    • Focus of study
    • Country of study
    • Variables being measured
    • Age range of participants
    • Location of the study
  • Study Research Questions
    • hypothesis
    • theoretical/empirical basis
  • Methods designs
    • Independent variables
    • Outcome variables
    • Measurement tools
  • Methods groups
    • Nonrandomized with treatment and control groups/repeated measures design
    • Number of groups
  • Methods sampling strategy
    • Explicitly stated/Implicit/not stated/unclear
    • sampling frame (telephone directory, electoral register, postcode, school listing)random selection/systematically/convenience
  • Sample information
    • number of participants in the study
    • if more than one group, the number of participants in each group
    • sex
    • socioeconomic status ethnicity
    • special educational need
    • region
    • control for bias from confounding variables and groups
    • baseline value for longitudinal study
  • Recruitment and consent
    • Method: letters of invitation, telephone, face-to-face
    • incentives
    • consent sought
  • Data collection
    • Methods: experimental, curriculum-based assessment, focus group, group interview, one-to-one interview, observation, self-completion questionnaire, self-completion report or diary, exams, clinical test, practical test, psychological test, school records, secondary data etc.
    • who collected the data
    • reliability
    • validity
  • Data analysis
    • statistical methods: descriptive, correlation, group differences (t test, ANOVA), growth curve analysis/multilevel modeling(HLM), structural equation modeling(SEM), path analysis, regression
  • Results and conclusion
    • Group means, SD, N, estimated effect size, appropriate SD, F, t test, significance, inverse variance weight

D. Statistics

  • Cohen’s kappa
  • Cohen’s d
  • effect size
  • aggregate/weighted mean effect size
  • 95% confidence interval: upper and lower
  • homogeneity of variance (Q statistic): Test if the mean effect size of the studies are significantly heterogeneous (p<.05), which means that there is more variability in the effect sizes than would be expected from sampling error and that the effect sized did not estimate common population mean (Lipsey & Wilson, 2001)
  • df: degrees of freedom
  • I square (%): the percentage of variability of the effect size that is attributable to true heterogeneity, that is, over and above the sampling error.
  • Outlier detection
  • mixed-effects model (consider studies as random effects): moderator analysis for heterogeneity (allow for population parameters to vary across studies, reducing the probability of committing a Type I error)
  • Proc GLM/ANOVA (consider studies as fixed effects): moderator analysis for heterogeneity
    • Region
    • Socioeconomic status
    • Geographical location
    • Education level
    • Setting
    • Language
    • sampling method
  • Statistical difference in the mean effect size of methodological feature of the study
    • confidence in effect size derivation (medium, high)
    • reliability (not reported, reported)
    • validity (not reported vs. reported
  • classic fail-safe N/Orwin’s fail-safe N: The number of missing null studies needed to bring the current mean effect size of the meta-analysis to .04. Threshhold is 5k+10, k is number of studies for the meta-analysis. If the N is greater than the 5k+10 limit then it is unlikely that publication bias poses a significant threat to the validity of findings of the meta-analysis.
    • Used to assess publication bias. eg. control for bias in studies (tightly controlled, loosely controlled, not controlled)

E. Purpose/Research Questions

  • Whether the treatment is associated with single effect or multiple effects?
  • Understand the variability of studies on the association of treatment with single or multiple effects, and explain the variable effects potentially through the study features (moderators). How do the effects of the treatment vary different study features?

F. Reference