BACKGROUND AND OBJECTIVE: This article explores the need for conceptual advances and practical guidance in the application of the GRADE approach within public health contexts. METHODS: We convened an expert workshop and conducted a scoping review to identify challenges experienced by GRADE users in public health contexts. We developed this concept article through thematic analysis and an iterative process of consultation and discussion conducted with members electronically and at three GRADE Working Group meetings. RESULTS: Five priority issues can pose challenges for public health guideline developers and systematic reviewers when applying GRADE: (1) incorporating the perspectives of diverse stakeholders; (2) selecting and prioritizing health and "nonhealth" outcomes; (3) interpreting outcomes and identifying a threshold for decision-making; (4) assessing certainty of evidence from diverse sources, including nonrandomized studies; and (5) addressing implications for decision makers, including concerns about conditional recommendations. We illustrate these challenges with examples from public health guidelines and systematic reviews, identifying gaps where conceptual advances may facilitate the consistent application or further development of the methodology and provide solutions. CONCLUSION: The GRADE Public Health Group will respond to these challenges with solutions that are coherent with existing guidance and can be consistently implemented across public health decision-making contexts.

Center for Evidence based Policy Oregon Health and Science University Portland OR 97201 USA

Cochrane Public Health and Health Systems Network University of Oxford Oxford UK

Department for Health Services Research Institute of Public Health and Nursing Research University of Bremen Grazer Straße 4 28359 Bremen Germany; Health Sciences Bremen University of Bremen 28359 Bremen Germany

Department of Clinical Epidemiology Amsterdam University Medical Centres University of Amsterdam Biostatistics and Bioinformatics Amsterdam Public Health Research Institute Amsterdam the Netherlands

Department of Health Research Methods Evidence and Impact McMaster University 1280 Main Street W Hamilton Ontario L8S 4K1 Canada

Department of Health Research Methods Evidence and Impact McMaster University 1280 Main Street W Hamilton Ontario L8S 4K1 Canada; Department of Internal Medicine American University of Beirut Beirut Lebanon

Department of Health Research Methods Evidence and Impact McMaster University 1280 Main Street W Hamilton Ontario L8S 4K1 Canada; Departments of Medicine and Biomedical and Health Informatics University of Missouri Kansas City Kansas City MO 66160 USA

Department of Health Research Methods Michael G DeGroote Cochrane Canada and McMaster GRADE Centres and WHO Collaborating Centre for Infectious Diseases Research Methods and Recommendations Evidence and Impact McMaster University Hamilton Ontario Canada

Department of Nutrition Texas A and M University College Station TX USA

Department of Paediatrics Obstetrics and Gynaecology and Preventative Medicine Universitat Autònoma de Barcelona; Faculty of Health Sciences Universidad Francisco de Vitoria ; CIBER Epidemiology and Public Health; Cochrane Associate Centre of Madrid Madrid Spain

European Commission Joint Research Centre Ispra Italy

Faculty of Medicine Czech National Centre for Evidence Based Healthcare and Knowledge Translation Institute of Biostatistics and Analyses Masaryk University 625 00 Brno Czechia

Institute for Medical Informatics Biometry and Epidemiology Pettenkofer School of Public Health LMU Munich Munich Germany

MRC CSO Social and Public Health Sciences Unit Berkeley Square 99 Berkeley Street University of Glasgow Glasgow G3 7HR UK

National Institute for Health and Care Excellence Level 1A City Tower Piccadilly Plaza Manchester M1 4BT UK

The Department of Vector Biology Partnership for Increasing the Impact of Vector Control Liverpool School of Tropical Medicine Liverpool UK

Zobrazit více v PubMed

Alonso-Coello P., Schünemann H.J., Moberg J., Brignardello-Petersen R., Akl E.A., Davoli M. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ. 2016:353. PubMed

Moberg J., Oxman A.D., Rosenbaum S., Schünemann H.J., Guyatt G., Flottorp S. The GRADE Evidence to Decision (EtD) framework for health system and public health decisions. Health Res Policy Syst. 2018;16(1):45. PubMed PMC

Schünemann H.J., Mustafa R., Brozek J., Santesso N., Alonso-Coello P., Guyatt G. GRADE Guidelines: 16. GRADE evidence to decision frameworks for tests in clinical practice and public health. J Clin Epidemiol. 2016;76:89–98. PubMed

Acheson D. The Stationery Office; London: 1988. Public Health in England: The Report of the Committee of Inquiry into the Future Development to the Public Health Function.

Griffiths S., Jewell T., Donnelly P. Public health in practice: the three domains of public health. Public Health. 2005;119(10):907–913. PubMed PMC

Benach J., Malmusi D., Yasui Y., Martínez J.M. A new typology of policies to tackle health inequalities and scenarios of impact based on Rose's population approach. J Epidemiol Community Health. 2013;67:286–291. PubMed

Ahmed F., Temte J.L., Campos-Outcalt D., Schünemann H.J., Recommendation A.E.B., Group A.E.B.R.W. Methods for developing evidence-based recommendations by the advisory committee on immunization practices (ACIP) of the U.S. Centers for disease control and prevention (CDC) Vaccine. 2011;29(49):9171–9176. PubMed

Dahm P., Oxman A.D., Djulbegovic B., Guyatt G.H., Murad M.H., Amato L. Stakeholders apply the GRADE evidence-to-decision framework to facilitate coverage decisions. J Clin Epidemiol. 2017;86:129–139. PubMed

Schunemann H.J., Hill S.R., Kakad M., Vist G.E., Bellamy R., Stockman L. Transparent development of the WHO rapid advice guidelines. PLoS Med. 2007;4(5):786–793. PubMed PMC

Schunemann H.J., Lerda D., Dimitrova N., Alonso-Coello P., Grawingholt A., Quinn C. Methods for development of the European commission initiative on Breast cancer guidelines recommendations in the era of guideline transparency. Ann Intern Med. 2019;171(4):273–280. PubMed

Thornton J., Alderson P., Tan T., Turner C., Latchem S., Shaw E. Introducing GRADE across the NICE clinical guideline program. J Clin Epidemiol. 2013;66:124–131. PubMed

Akl E.A., Kennedy C., Konda K., Caceres C.F., Horvath T., Ayala G. Using GRADE methodology for the development of public health guidelines for the prevention and treatment of HIV and other STIs among men who have sex with men and transgender people. BMC Public Health. 2012;12:386. PubMed PMC

Alexander P.E., Gionfriddo M.R., Li S.A., Bero L., Stoltzfus R.J., Neumann I. A number of factors explain why WHO guideline developers make strong recommendations inconsistent with GRADE guidance. J Clin Epidemiol. 2016;70:111–122. PubMed

Rehfuess E.A., Akl E.A. Current experience with applying the GRADE approach to public health interventions: an empirical study. BMC Public Health. 2013;13:9. PubMed PMC

Dedios M.C., Esperato A., De-Regil L.M., Pena-Rosas J.P., Norris S.L. Improving the adaptability of WHO evidence-informed guidelines for nutrition actions: results of a mixed methods evaluation. Implementation Sci. 2017;12(1):39. PubMed PMC

European Centre for Disease Prevention and Control Evidence-based methodologies for public health [Technical report] 2011. https://ecdc.europa.eu/en/publications-data/evidence-based-methodologies-public-health Available at.

Harder T., Abu Sin M., Bosch-Capblanch X., Bruno C., de Carvalho Gomes H., Duclos P. Towards a framework for evaluating and grading evidence in public health. Health Policy. 2015;119(6):732–736. PubMed

Thayer K.A., Schünemann H.J. Using GRADE to respond to health questions with different levels of urgency. Environ Int. 2016;92-93:585–589. PubMed

Florez I.D., Morgan R.L., Falavigna M., Kowalski S.C., Zhang Y., Etxeandia-Ikobaltzeta I. Development of rapid guidelines: 2. A qualitative study with WHO guideline developers. Health Res Policy Syst. 2018;16(1):62–69. PubMed PMC

Schünemann H., Hill S., Guyatt G., Akl E.A., Ahmed F. The GRADE approach and Bradford Hill's criteria for causation. J Epidemiol Community Health. 2011;65:392–395. PubMed

Akl E.A., Welch V., Pottie K., Eslava-Schmalbach J., Darzi A., Sola I. GRADE equity guidelines 2: considering health equity in GRADE guideline development: equity extension of the guideline development checklist. J Clin Epidemiol. 2017;90:68–75. PubMed PMC

Cuello-Garcia C.A., Morgan R.L., Brozek J., Santesso N., Verbeek J., Thayer K. A scoping review and survey provides the rationale, perceptions, and preferences for the integration of randomized and nonrandomized studies in evidence syntheses and GRADE assessments. J Clin Epidemiol. 2018;98:33–40. PubMed

Morgan R.L., Thayer K.A., Bero L., Bruce N., Falck-Ytter Y., Ghersi D. GRADE: assessing the quality of evidence in environmental and occupational health. Environ Int. 2016;92-93:611–616. PubMed PMC

Pottie K., Welch V., Morton R., Akl E.A., Eslava-Schmalbach J.H., Katikireddi V. GRADE equity guidelines 4: considering health equity in GRADE guideline development: evidence to decision process. J Clin Epidemiol. 2017;90:84–91. PubMed PMC

Schünemann H.J., Cuello C., Akl E.A., Mustafa R.A., Meerpohl J.J., Thayer K. GRADE guidelines: 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence. J Clin Epidemiol. 2019;111:105–114. PubMed PMC

Welch V.A., Akl E.A., Guyatt G., Pottie K., Eslava-Schmalbach J., Ansari M.T. GRADE equity guidelines 1: considering health equity in GRADE guideline development: introduction and rationale. J Clin Epidemiol. 2017;90:59–67. PubMed PMC

Welch V.A., Akl E.A., Pottie K., Ansari M.T., Briel M., Christensen R. GRADE equity guidelines 3: considering health equity in GRADE guideline development: rating the certainty of synthesized evidence. J Clin Epidemiol. 2017;90:76–83. PubMed PMC

Alexander P.E., Li S.-A., Gionfriddo M.R., Stoltzfus R.J., Neumann I., Brito J.P. Senior GRADE methodologists encounter challenges as part of WHO guideline development panels: an inductive content analysis. J Clin Epidemiol. 2016;70:123–128. PubMed

Burford B.J., Rehfuess E., Schunemann H.J., Akl E.A., Waters E., Armstrong R. Assessing evidence in public health: the added value of GRADE. J Public Health. 2012;34(4):631–635. PubMed

Gopinathan U., Hoffman S.J. Institutionalising an evidence-informed approach to guideline development: progress and challenges at the World Health Organization. BMJ Glob Health. 2018;3(5):e000716. PubMed PMC

Harder T., Abu Sin M., Bosch-Capblanch X., Bruno C., de Carvalho Gomes H., Duclos P. Towards a framework for evaluating and grading evidence in public health. Health Policy. 2015;119(6):732–736. PubMed

Sterne J.A.C., Hernan M.A., Reeves B.C., Savovic J., Berkman N.D., Viswanathan M. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919. PubMed PMC

Lewin S., Glenton C., Munthe-Kaas H., Carlsen B., Colvin C.J., Gulmezoglu M. Using qualitative evidence in decision making for health and social interventions: an approach to assess confidence in findings from qualitative evidence syntheses (GRADE-CERQual) PLoS Med. 2015;12(10):e1001895. PubMed PMC

Murad M.H., Mustafa R.A., Schünemann H.J., Sultan S., Santesso N. Rating the certainty in evidence in the absence of a single estimate of effect. Evid Based Med. 2017;22(3):85–87. PubMed PMC

Juarez S.P., Honkaniemi H., Dunlavy A.C., Aldridge R.W., Barreto M.L., Katikireddi S.V. Effects of non-health-targeted policies on migrant health: a systematic review and meta-analysis. Lancet Glob Health. 2019;7(4):e420–e435. PubMed PMC

National Institute for Health and Care Excellence Physical activity and the environment. NICE guideline [NG90] 2018. https://www.nice.org.uk/guidance/ng90 Available at.

Stone D. WW Norton; New York: 1997. Policy paradox: The art of political decision making.

Lorenc T., Tyner E.F., Petticrew M., Duffy S., Martineau F.P., Phillips G. Cultures of evidence across policy sectors: systematic review of qualitative evidence. Eur J Public Health. 2014;24:1041–1047. PubMed

Griffiths S.M. The Sustainable Development Goals: an agenda for us all. Perspect Public Health. 2019;139(5):224–225. PubMed

Morton S., Pencheon D., Squires N. Sustainable Development Goals (SDGs), and their implementation: a national global framework for health, development and equity needs a systems approach at every level. Br Med Bull. 2017;124(1):81–90. PubMed

Rehfuess E.A., Stratil J.M., Scheel I.B., Portela A., Norris S.L., Baltussen R. The WHO-INTEGRATE evidence to decision framework version 1.0: integrating WHO norms and values and a complexity perspective. BMJ Glob Health. 2019;4(Suppl 1):e000844. PubMed PMC

Rose G. Strategy of prevention: lessons from cardiovascular disease. BMJ. 1981;282:1847. PubMed PMC

Webster J., Waqanivalu T., Arcand J., Trieu K., Cappuccio F.P., Appel L.J. Understanding the science that supports population-wide salt reduction programs. J Clin Hypertens (Greenwich, Conn) 2017;19(6):569–576. PubMed PMC

Petticrew M., McKee M., Lock K., Green J., Phillips G. In search of social equipoise. BMJ. 2013;347:f4016. PubMed

Craig P., Katikireddi S.V., Leyland A., Popham F. Natural experiments: an overview of methods, approaches, and contributions to public health intervention research. Annu Rev Public Health. 2017;38(1):39–56. PubMed PMC

von Philipsborn P., Stratil J.M., Burns J., Busert L.K., Pfadenhauer L.M., Polus S. Environmental interventions to reduce the consumption of sugar-sweetened beverages and their effects on health. Cochrane Database Syst Rev. 2019;6:Cd012292. PubMed PMC

Burns J., Boogaard H., Polus S., Pfadenhauer L.M., Rohwer A.C., van Erp A.M. Interventions to reduce ambient particulate matter air pollution and their effect on health. Cochrane Database Syst Rev. 2019;5:Cd010919. PubMed PMC

Morgan R.L., Thayer K.A., Santesso N., Holloway A.C., Blain R., Eftim S.E. A risk of bias instrument for non-randomized studies of exposures: a users' guide to its application in the context of GRADE. Environ Int. 2019;122:168–184. PubMed PMC

Andrews J.C., Schünemann H.J., Oxman A.D., Pottie K., Meerpohl J.J., Coello P.A. GRADE guidelines: 15. Going from evidence to recommendation—determinants of a recommendation's direction and strength. J Clin Epidemiol. 2013;66:726–735. PubMed

Schunemann H.J., Hill S.R., Kakad M., Bellamy R., Uyeki T.M., Hayden F.G. WHO rapid advice guidelines for pharmacological management of sporadic human infection with avian influenza A (H5N1) virus. Lancet Infect Dis. 2007;7:21–31. PubMed PMC

Fischer A.J., Threlfall A., Meah S., Cookson R., Rutter H., Kelly M.P. The appraisal of public health interventions: an overview. J Public Health. 2013;35(4):488–494. PubMed

Hultcrantz M., Rind D., Akl E.A., Treweek S., Mustafa R.A., Iorio A. The GRADE Working Group clarifies the construct of certainty of evidence. J Clin Epidemiol. 2017;87:4–13. PubMed PMC

Threlfall A.G., Meah S., Fischer A.J., Cookson R., Rutter H., Kelly M.P. The appraisal of public health interventions: the use of theory. J Public Health. 2015;37(1):166–171. PubMed

Pega F., Liu S.Y., Walter S., Lhachimi S.K. Unconditional cash transfers for assistance in humanitarian disasters: effect on use of health services and health outcomes in low- and middle-income countries. Cochrane Database Syst Rev. 2015;(9):CD011247. PubMed PMC

Pega F., Liu S.Y., Walter S., Pabayo R., Saith R., Lhachimi S.K. Unconditional cash transfers for reducing poverty and vulnerabilities: effect on use of health services and health outcomes in low- and middle-income countries. Cochrane Database Syst Rev. 2017;(11):CD011135. PubMed PMC

World Health Organization Guidelines for malaria vector control. 2019. https://www.who.int/malaria/publications/atoz/9789241550499/en/ Available at. PubMed

Choi L., Pryce J., Garner P. The combination of indoor residual spraying with insecticide-treated nets versus insecticide-treated nets alone for preventing malaria. Cochrane Database Syst Rev. 2017;(6):CD012688. PubMed PMC

Choi L., Wilson A. Larviciding to control malaria. Cochrane Database Syst Rev. 2017;(7):CD012736. PubMed PMC

Gleave K., Lissenden N., Richardson M., Choi L., Ranson H. Piperonyl butoxide (PBO) combined with pyrethroids in insecticide-treated nets to prevent malaria in Africa. Cochrane Database Syst Rev. 2018;11:CD012776. PubMed PMC

Maia M.F., Kliner M., Richardson M., Lengeler C., Moore S.J. Mosquito repellents for malaria prevention. Cochrane Database Syst Rev. 2018;2:Cd011595. PubMed PMC

Pluess B., Tanser F.C., Lengeler C., Sharp B.L. Indoor residual spraying for preventing malaria. Cochrane Database Syst Rev. 2010:Cd006657. PubMed PMC

Pryce J., Choi L., Richardson M., Malone D. Insecticide space spraying for preventing malaria transmission. Cochrane Database Syst Rev. 2018;11:Cd012689. PubMed PMC

Pryce J., Richardson M., Lengeler C. Insecticide-treated nets for preventing malaria. Cochrane Database Syst Rev. 2018;(11):CD000363. PubMed PMC

Walshe D.P., Garner P., Adeel A.A., Pyke G.H., Burkot T.R. Larvivorous fish for preventing malaria transmission. Cochrane Database Syst Rev. 2017;(12):CD008090. PubMed PMC

European Commission JRC in brief 2019. https://ec.europa.eu/jrc/en/about/jrc-in-brief Available at.

European Commission European Breast Guidelines 2019. https://ecibc.jrc.ec.europa.eu/recommendations/ Available at.

European Commission Methods of the European guidelines for breast cancer screening and diagnosis 2016. https://ecibc.jrc.ec.europa.eu/documents/20181/82470/GDG+Methodology/a8fb475b-b5c6-474f-80f2-3ae8b9c27461 Available at.

European Commission Scope of the European guidelines for breast cancer screening and diagnosis. 2017. https://ecibc.jrc.ec.europa.eu/-/scope-of-the-european-guidelines-for-breast-cancer-screening-and-diagnos-1 Available at.

Schünemann H.J., Lerda D., Quinn C., Follmann M., Alonso-Coello P., Rossi P.G. Breast cancer screening and diagnosis: a synopsis of the European Breast guidelines. Ann Intern Med. 2020;172:46–56. PubMed

Montgomery P., Movsisyan A., Grant S.P., Macdonald G., Rehfuess E.A. Considerations of complexity in rating certainty of evidence in systematic reviews: a primer on using the GRADE approach in global health. BMJ Glob Health. 2019;4(Suppl 1):e000848. PubMed PMC

Addressing Challenges When Applying GRADE to Public Health Guidelines: A Scoping Review Protocol and Pilot Analysis