Open Pedagogy/STEM Librarian
Evidence Synthesis is a type of research activity that uses articles/papers as its data set.
- Systematic Reviews are a type of evidence synthesis.
- Differences in type come from both level of rigor and specificity of question.
Types of Reviews
Systematic Review is a methodical and comprehensive literature synthesis focused on a well-formulated research question.
- Should take 2-5 people approximately 12-18 months to complete.
- Aims to identify and synthesize all of the scholarly research on a particular topic, including both published and unpublished studies.
- Conducted in an unbiased, reproducible way to provide evidence for practice and policy-making.
- May involve a meta-analysis.
- Much more time-intensive than traditional literature reviews.
Steps for a systematic review study:
A systematic review involves a number of iterative processes, including gathering, analyzing, integrating (data of findings), and synthesizing and explaining all parts of the process (results). This procedure could potentially be broken down into the following smaller steps:
- Define the research question
- Develop a search protocol
- Run searches
- Retrieve and de-duplicate references
- Screen studies against inclusion/exclusion criteria
- Extract data and assess studies for risk of bias
- Clean and manage data
- Conduct qualitative synthesis
- Conduct meta-analysis if appropriate
- Write a report
To maintain transparency, the procedure must be fully explained and described in the research report's final version.
Recommended Systematic Review Methods Readings
- An introduction to systematic reviews, Gough et al. (2017)
- ROBIS: Risk of Bias in Systematic Reviews
- Kmet, L. M., Lee, R. C., Cook, L. S., & Alberta Heritage Foundation for Medical Research. (2004). Standard quality assessment criteria for evaluating primary research papers from a variety of fields. Edmonton, Alta: Alberta Heritage Foundation for Medical Research.
- Acosta, S., Garza, T., Hsu, H. Y., & Goodson, P. (2020). Assessing quality in systematic literature reviews: A study of novice rater training. Sage Open, 10(3), 2158244020939530.
Examples of systematic review studies:
Martin, J. P., Choe, N. H., Halter, J., Foster, M., Froyd, J., Borrego, M., & Winterer, E. R. (2019). Interventions supporting baccalaureate achievement of Latinx STEM students matriculating at 2‐year institutions: A systematic review. Journal of Research in Science Teaching, 56(4), 440-464.
BuHamdan, S., Alwisy, A., & Bouferguene, A. (2021). Generative systems in the architecture, engineering and construction industry: A systematic review and analysis. International Journal of Architectural Computing, 19(3), 226-249.
Scoping Reviews is a method that systematically and transparently collects and categorizes existing evidence on a broad topic.
- Seeks to identify research gaps and opportunities for evidence synthesis.
- May critically evaluate existing evidence, but does not attempt to synthesize the results in the way a systematic review would.
- May take longer than a systematic review.
- See Scoping studies: towards a methodological framework.
- See Environmental Evidence Journal Systematic Maps and Guidance on Systematic Maps—CIFOR.
Scoping Review Steps
- Develop the research question(s)
- Develop the inclusion criteria
- Develop a comprehensive search strategy
- Search relevant databases
- Perform grey literature searches (dissertations, conference papers, reports) and supplementary searches (e.g., forward/backwards citations, handsearching journals, contacting authors)
- Screen titles and abstracts using eligibility criteria, usually done by 2 reviewers
- Screen full text noting the reason for exclusion, usually done by 2 reviewers
- Develop a form or table to extract (or chart) data from the study
- Analyze data, typically using descriptive statistics or descriptive approaches
- Report search methods; and report results of the literature searches and screening (PRISMA Flow Chart/Diagram)
- Report results that answer the study's research questions
Based on Arksey & O'Malley (2005) and Peters et al. (2020).
Other resources: Steps for conducting a scoping review.
Recommended Scoping Review Methods Readings
- Scoping studies: towards a methodological framework, Arksey & O'Malley (2005)
- Scoping Reviews Chapter in JBI Manual for Evidence Synthesis, Peters et al. (2020)
- Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach, Munn et al. (2018)
- An introduction to scoping reviews, Mak & Thomas (2022)
Example:s of scoping review studies:
Xu, Z., Zhou, X., Kogut, A., & Watts, J. (2022). A scoping review: Synthesizing evidence on data management instruction in academic libraries. The Journal of Academic Librarianship, 48(3), 102508.
Lamanna M, Klinger CA, Liu A, Mirza RM. The association between public transportation and social isolation in older adults: A scoping review of the literature. Canadian Journal on Aging. 2020;39(3):393-405. doi:10.1017/S0714980819000345
Meta-analysis may be conducted independently or as part of a systematic review. it is a method for systematically combining pertinent qualitative and quantitative study data from several selected studies to develop a single conclusion that has greater statistical power. This conclusion is statistically stronger than the analysis of any single study, due to increased numbers of subjects, greater diversity among subjects, or accumulated effects and results. It uses statistical methods to objectively evaluate, synthesize, and summarize results.
In a meta-analysis, the results of each study are translated to a common metric, in terms of an effect-size index, and they are put into relation with study-level moderator variables in order to explain the variability among the effect estimates.
Meta-analysis would be used for the following purposes:
- To establish statistical significance with studies that have conflicting results
- To develop a more correct estimate of effect magnitude
- To provide a more complex analysis of harms, safety data, and benefits
- To examine subgroups with individual numbers that are not statistically significant
Advantages
- Greater statistical power
- Confirmatory data analysis
- Greater ability to extrapolate to general population affected
- Considered an evidence-based resource
Disadvantages
- Difficult and time consuming to identify appropriate studies
- Not all studies provide adequate data for inclusion and analysis
- Requires advanced statistical techniques
- Heterogeneity of study populations
Click here to learn more.
Recommended Meta-analysis Methods Readings
- Introduction to Meta-Analysis: A Guide for the Novice, Wilson (2014)
- Doing Meta-Analysis with R: A Hands-On Guide, Harrer et al. (2021)
Example of meta-analysis studies:
Aldao, A., Nolen-Hoeksema, S., & Schweizer, S. (2010). Emotion-regulation strategies across psychopathology: A meta-analytic review. Clinical Psychology Review, 30(2), 217–237. https://doi-org.ezproxy.lib.uh.edu/10.1016/j.cpr.2009.11.004
Zhou, X., Shu, L., Xu, Z., & Padrón, Y. (2023). The effect of professional development on in-service STEM teachers’ self-efficacy: a meta-analysis of experimental studies. International Journal of STEM Education, 10(1), 1-20. https://doi.org/10.1186/s40594-023-00422-x
Rapid Review
Applies systematic review methodology within a time-constrained setting.
- Employs methodological “shortcuts” (limiting search terms for example) at the risk of introducing bias.
- Useful for addressing issues needing quick decisions.
- See Evidence summaries: the evolution of a rapid review approach for methodological guidance.
Umbrella Review
Reviews other systematic reviews on a topic.
- Often defines a broader question than is typical of a traditional systematic review.
- Most useful when there are competing interventions to consider.
- See Umbrella Reviews in JBI Manual for Evidence Synthesis
Source: This work was created by Cornell University Library and is licensed under a Creative Commons Attribution 4.0 International License. 