LibGuides: Systematic Reviews/Evidence Synthesis: Quality Assessment/Risk of Bias Tools

Studies included in a systematic review need to be evaluated for Risk of Bias. Per Page et al., (2021) "Risk of bias refers to the potential for study findings to systematically deviate from the truth due to methodological flaws in the design, conduct or analysis." Risk of Bias tools are not measuring bias, but rather the risk of bias inherent in a given study. See Reducing Bias.

Repository of Quality Assessment and Risk of Bias Tools
A handy resource from Duke University' Medical Center Library for finding and selecting a risk of bias or quality assessment tool for evidence synthesis projects. Download spreadsheet for full functionality.

Hint: Sort the spreadsheet by study type to find the appropriate risk of bias tool for your study:

Instructions for Use

This project was created and is maintained by Duke University's Medical Center Library & Archives. Quality Assessment and Risk of Bias Tool Repository © 2021 by Leila Ledbetter and Stephanie Hendren is licensed under Attribution-NonCommercial 4.0 International.To view a copy of this license, visit http://creativecommons.org/licenses/by-nc/4.0/

Some common Risk of Bias Tools:

Downs and Black
See Appendix. Provides both an overall score for study quality and a numeric score out of a possible 30 points. It has five sections. Administration of the tool can happen either within a systematic review process, or as a quality assessment tool for individual articles. (Downs & Black, 1998).

JBI Critical Appraisal Tools Joanna Briggs Institute (JBI) is an independent, international, not-for-profit researching and development organization based at the University of Adelaide, South Australia. Includes a number of critical appraisal tools including Checklist for Randomized Controlled Trials.

Modified Downs and Black
See Appendix 4. (MacLehose et al., 2000).
more...less...
In their cited paper, they state they made modifications to quantify "the relationship between study quality and effect size.Modifications were made, or additional questions developed, to quantify the extent to which factors likely to influence effect size (e.g. adjustment for confounders, resolution with which confounding variables were measured) were addressed by a study. The original questions about external validity were also merged into a single question. By assessing all the studies on the same question about external validity, we sought to obtain some indication of the external validity of different study designs and hence the extent to which differences in effect size could be attributed to this factor."
Newcastle Ottawa Scale
For nonrandomized studies including case control or cohort studies.
RoB 2.0
A revised Cochrane risk of bias tool for randomized trials. Suitable for individually-randomized, parallel-group, and cluster- randomized trials
ROB-ME
A tool for assessing Risk Of Bias due to Missing Evidence in a synthesis.
Matthew Page, Monash University. Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
robvis
A web app designed to for visualizing risk-of-bias assessments performed as part of a systematic review. The tool creates “traffic light” plots of the domain-level judgements for each individual result; and weighted bar plots of the distribution of risk-of-bias judgements within each bias domain.

Tools for Specific Study Types

Integrative Reviews

Critical Appraisal Skills Programme (CASP) Checklists Appraisal checklists designed for use with Systematic Reviews, Randomized Controlled Trials, Cohort Studies, Case Control Studies, Economic Evaluations, Diagnostic Studies, Qualitative studies and Clinical Prediction Rule.
Mixed Methods Appraisal Tool (MMAT) The MMAT is a critical appraisal tool that is designed for the appraisal stage of systematic mixed studies reviews, i.e., reviews that include qualitative, quantitative and mixed methods studies. It permits to appraise the methodological quality of five categories to studies: qualitative research, randomized controlled trials, non-randomized studies, quantitative descriptive studies, and mixed methods studies. (Hong et al., 2018).

Randomized Controlled Trials

CASP Checklists Critical Assessment Skills Programme (CASP) has appraisal checklists designed for use with Systematic Reviews, Randomized Controlled Trials, Cohort Studies, Case Control Studies, Economic Evaluations, Diagnostic Studies, Qualitative studies and Clinical Prediction Rule.
JBI Critical appraisal Tools Joanna Briggs Institute (JBI) is an independent, international, not-for-profit researching and development organization based at the University of Adelaide, South Australia. Contains a number of critical appraisal tools including Checklist for Randomized Controlled Trials
RoB 2.0 A revised Cochrane risk of bias tool for randomized trials. Is suitable for individually-randomized, parallel-group, and cluster- randomized trials

Qualitative Studies

CASP Qualitative Research Checklist Most frequently recommended tool for qualitative study assessment
JBI Critical Appraisal Checklist for Qualitative Research

Systematic Reviews

Scoping and Other Review Types

CAT HPPR: A critical appraisal tool to assess the quality of systematic, rapid, and scoping reviews investigating interventions in health promotion and prevention (Heise et al., 2022).

CAT HPPR
Critical Appraisal Tool for Health Promotion and Prevention Reviews
CAT HPPR Manual and Instructions
Manual and instructions to reviewers for using the Critical Appraisal Tool for
Health Promotion and Prevention Reviews (CAT HPPR). 2020.

References:

Buccheri, R. K., & Sharifi, C. (2017). Critical appraisal tools and reporting guidelines for evidence‐basedpPractice. Worldviews on Evidence-Based Nursing, 14(6), 463–472. https://doi.org/10.1111/wvn.12258

Burls, A. (2009). What is critical appraisal? Retrieved April 21, 2022, from www.whatisseries.co.uk

Downs, S. H., & Black, N. (1998). The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. Journal of Epidemiology and Community Health (1979), 52(6), 377–384. https://doi.org/10.1136/jech.52.6.377

Downs and Black Checklist for Clinical Trial Quality Assessment.(2013). In Point-of-Care Testing of International Normalized Ratio for Patients on Oral Anticoagulant Therapy – Project Protocol [Internet]. Canadian Agency for Drugs and Technologies in Health. https://www.ncbi.nlm.nih.gov/books/NBK361373/

Heise, T. L., Seidler, A., Girbig, M., Freiberg, A., Alayli, A., Fischer, M., Haß, W., & Zeeb, H. (2022). CAT HPPR: A critical appraisal tool to assess the quality of systematic, rapid, and scoping reviews investigating interventions in health promotion and prevention. BMC Medical Research Methodology, 22(1), 334–334. https://doi.org/10.1186/s12874-022-01821-4

Hong, Q.N., Fàbregues, S., Bartlett, G., Boardman, F.K., Cargo, M., Dagenais, P., Gagnon, M., Griffiths, F.E., Nicolau, B., O’Cathain, A., Rousseau, M.C., Vedel, I., & Pluye, P. (2018). The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers. Education for Information, 34(4), 285-291.DOI 10.3233/EFI-180221

Ma, Wang, Y., Yang, Z., Huang, D., Weng, H., & Zeng, X. (2020). Methodological quality (risk of bias) assessment tools for primary and secondary medical studies: what are they and which is better? Military Medical Research, 7(1), 7.

MacLehose RR, Reeves BC, Harvey IM, Sheldon TA, Russell IT, Black AM. (2000). A systematic review of comparisons of effect sizes derived from randomised and non-randomised studies. Health Technology Assessment,;4(34),:1-154.

Motheral, B., Brooks, J., Clark, M. A., Crown, W. H., Davey, P., Hutchins, D., Martin, B. C., & Stang, P. (2003). A checklist for retrospective database studies—Report of the ISPOR task force on retrospective databases. Value in Health, 6(2), 90–97. https://doi.org/10.1046/j.1524-4733.2003.00242.x

Page, M. J., Moher, D., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., McGuinness, L. A., … McKenzie, J. E. (2021). PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ (Clinical research ed.), 372, n160. https://doi-org.adelphi.idm.oclc.org/10.1136/bmj.n160

Shea, B. J., Reeves, B. C., Wells, G., Thuku, M., Hamel, C., Moran, J., Moher, D., Tugwell, P., Welch, V., Kristjansson, E., & Henry, D. A. (2017). AMSTAR 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ (Clinical research ed.), 358, j4008. https://doi.org/10.1136/bmj.j4008