Skip to Main Content

Critical Appraisal of Studies

Critical Appraisal

Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context (Burls, 2009). Critical appraisal of studies involves checking the quality, reliability and relevance of the studies you've selected to help answer your review question. Depending on the type of study you are evaluating you may use different evaluation tools. When evaluating studies, some questions to consider are:

  • ​​​​Has the study's aim been clearly stated?
  • Does the sample accurately reflect the population?
  • Has the sampling method and size been described and justified?
  • Have exclusions been stated?
  • Is the control group easily identified?
  • Is the loss to follow-up detailed?
  • Can the results be replicated?
  • Are there confounding factors?
  • Are the conclusions logical?
  • Can the results be extrapolated to other populations?

Adapted from: University of Illinois, Chicago Library

 

More on critical appraisal:

Critical Appraisal Tools

Tools for Specific Study Types

Integrative Reviews

  • Critical Appraisal Skills Programme (CASP) Checklists Appraisal checklists designed for use with Systematic Reviews, Randomized Controlled Trials, Cohort Studies, Case Control Studies, Economic Evaluations, Diagnostic Studies, Qualitative studies and Clinical Prediction Rule.
  • Mixed Methods Appraisal Tool (MMAT) The MMAT is a critical appraisal tool that is designed for the appraisal stage of systematic mixed studies reviews, i.e., reviews that include qualitative, quantitative and mixed methods studies. It permits to appraise the methodological quality of five categories to studies: qualitative research, randomized controlled trials, non-randomized studies, quantitative descriptive studies, and mixed methods studies. (Hong et al., 2018).

Randomized Controlled Trials

  • CASP checklist for RCT 
  • CASP Checklists Critical Assessment Skills Programme (CASP) has appraisal checklists designed for use with Systematic Reviews, Randomized Controlled Trials, Cohort Studies, Case Control Studies, Economic Evaluations, Diagnostic Studies, Qualitative studies and Clinical Prediction Rule.
  • JBI Critical Appraisal Tools Joanna Briggs Institute (JBI) is an independent, international, not-for-profit researching and development organization based at the University of Adelaide, South Australia. Contains a number of critical appraisal tools including Checklist for Randomized Controlled Trials
  • RoB 2.0  A revised Cochrane risk of bias tool for randomized trials. Is suitable for individually-randomized, parallel-group, and cluster- randomized trials

Qualitative Studies

Systematic Reviews

Scoping and Other Review Types

References:

Buccheri, R. K., & Sharifi, C. (2017). Critical appraisal tools and reporting guidelines for evidence‐basedpPractice. Worldviews on Evidence-Based Nursing, 14(6), 463–472. https://doi.org/10.1111/wvn.12258

Burls, A. (2009). What is critical appraisal? Retrieved April 21, 2022, from www.whatisseries.co.uk

Downs, S. H., & Black, N. (1998). The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. Journal of Epidemiology and Community Health (1979)52(6), 377–384. https://doi.org/10.1136/jech.52.6.377

Downs and Black Checklist for Clinical Trial Quality Assessment.(2013). In Point-of-Care Testing of International Normalized Ratio for Patients on Oral Anticoagulant Therapy – Project Protocol [Internet]. Canadian Agency for Drugs and Technologies in Health. https://www.ncbi.nlm.nih.gov/books/NBK361373/

Heise, T. L., Seidler, A., Girbig, M., Freiberg, A., Alayli, A., Fischer, M., Haß, W., & Zeeb, H. (2022). CAT HPPR: A critical appraisal tool to assess the quality of systematic, rapid, and scoping reviews investigating interventions in health promotion and prevention. BMC Medical Research Methodology22(1), 334–334. https://doi.org/10.1186/s12874-022-01821-4

Hong, Q.N., Fàbregues, S., Bartlett, G., Boardman, F.K., Cargo, M., Dagenais, P., Gagnon, M., Griffiths, F.E., Nicolau, B., O’Cathain, A., Rousseau, M.C., Vedel, I., & Pluye, P. (2018). The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers. Education for Information, 34(4), 285-291.DOI 10.3233/EFI-180221

Ma, Wang, Y., Yang, Z., Huang, D., Weng, H., & Zeng, X. (2020). Methodological quality (risk of bias) assessment tools for primary and secondary medical studies: what are they and which is better? Military Medical Research, 7(1), 7. 

Motheral, B., Brooks, J., Clark, M. A., Crown, W. H., Davey, P., Hutchins, D., Martin, B. C., & Stang, P. (2003). A checklist for retrospective database studies—Report of the ISPOR task force on retrospective databases. Value in Health6(2), 90–97. https://doi.org/10.1046/j.1524-4733.2003.00242.x

Shea, B. J., Reeves, B. C., Wells, G., Thuku, M., Hamel, C., Moran, J., Moher, D., Tugwell, P., Welch, V., Kristjansson, E., & Henry, D. A. (2017). AMSTAR 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ (Clinical research ed.)358, j4008. https://doi.org/10.1136/bmj.j4008

Tod, D., Booth, A., & Smith, B. (2021). Critical appraisal. International Review of Sport and Exercise Psychology, 15(1), 52-72.