Using a consensus-based approach to build a threat assessment tool

Abstract

Background: Public health intelligence activities, including assessment of emerging public health threats, are core operations for many public health agencies. Threat assessment is intended to guide public health actions proportional to the assessed threat level within the context of an evolving situation. A risk-based threat assessment framework helps clarify the overall population-level health impacts of threats, guiding appropriate public health action.

Objective: This work aimed to develop a standardized, risk-based tool for assessing public health threats that could be used across the Public Health Agency of Canada (PHAC), integrating diverse perspectives from various PHAC stakeholders to ensure the tool is applicable across the full entire spectrum of potential public health threats.

Methods: A threat assessment framework was developed to assign an overall threat level based on the assessment of three distinct threat attributes: the severity of harm to human health, the degree to which a threat is likely to impact Canada, and Canada’s capacity to prevent or mitigate the threat. A Delphi-based approach was used to develop a set of qualitative criteria enabling end-users to assign a rating of high, moderate, or low for each threat attribute and the overall assessment.

Results: Three iterative surveys and two meetings led to consensus on definitions for the three threat attributes and the overall assessment, enabling a flexible framework capable of characterizing a wide range of public health threats.

Conclusion: Standardized, broadly accepted definitions for threat assessment reliably characterize and communicate public health events of importance and provide support for public health action.

Introduction

In a global environment marked by an increasing number of high-impact public health events ^{Footnote 1}, public health systems capable of rapidly detecting and assessing the significance of potential threats are essential to support decision-making by public health officials. A public health threat can be defined as verified data indicating a potential acute risk to human health that has been assessed as having the potential to harm to the health of a population ^{Footnote 2Footnote 3}. Early detection and assessment of potential public health threats is intended to guide response planning and actions proportional to the assessed level of threat within the context of an evolving situation ^{Footnote 4}. Public health actions resulting from early assessment can limit the impact of a threat on human health.

Public health intelligence activities, including the early assessment of potential public health threats, are core operations of many national and international public health agencies ^{Footnote 5Footnote 6Footnote 7}. Threat assessment involves characterizing a public health threat to understand its overall severity and population health impact. There is limited information in the published and grey literature about the threat assessment methodology used by public health agencies. However, public-facing reports provide an indication of the elements considered in such assessments. These include potential public health impact, severity of health effects, likelihood of exposure or transmission, and existence of control measures ^{Footnote 1Footnote 5Footnote 6Footnote 7}.

At the Public Health Agency of Canada (PHAC), a formalized Coordinated Threat Assessment (CTA) process was established in May 2022 in response to recommendations from the Auditor General of Canada to strengthen PHAC’s risk assessment process following an audit related to the COVID-19 pandemic ^{Footnote 4}. The CTA process sits within a spectrum of risk assessment activities, ranging from preliminary threat assessments to full, in-depth risk assessments ^{Footnote 8}. Within this spectrum, CTA coordinates technical subject-matter experts and integrates data and information from multiple sources to rapidly and systematically assess emerging public health events. This process encourages coordination of activities across PHAC and results in assessments that are standardized across programs, providing PHAC senior leadership with an overview of events that may constitute a threat to Canada and potentially require further public health action.

The original PHAC threat assessment method was designed as a user-friendly algorithm to characterize threats based on the type and urgency of federal public health actions required, such as monitoring, further investigation, or action within 48 hours. After a full year of use, along with feedback from an internal evaluation, the need was identified for a method that better captures the overall population health impact of threats to more effectively guide public health action. This led to the decision in 2023 to revise the threat assessment method to a risk-based approach.

The goal of this work was to develop a standardized tool that could be used across PHAC programs to characterize public health threats using a risk-based framework with high, moderate, and low threat levels for Canada. To address the challenge of designing a common tool that supports threat assessment across diverse populations and hazard types, a Delphi-based approach was used to gather perspectives from a range of PHAC stakeholders, ensuring applicability across the full spectrum of potential public health threats. This article describes the Delphi consensus process and presents the resulting threat assessment tool.

Methods

Multiple steps were taken to develop and finalize the threat assessment tool. Figure 1 depicts the activities and timeline associated with the process. Preliminary work included internal discussions within the threat assessment team, an informal environmental scan, and engagement with internal and international colleagues.

Figure 1 - Text description

This figure describes the process by which the threat assessment framework was developed and finalized between July 2023 and May 2024. Preliminary work, including an informal environmental scan and engagement with international and internal Public Health Agency of Canada (PHAC) colleagues, resulted in a threat assessment framework consisting of three threat attributes and an overall assessment, each with three levels of assessment. Informal feedback about the framework was solicited from PHAC programs and appropriate modifications were made. Once the framework was developed, a Delphi-based approach was used achieve at least 80% consensus on the definitions for each threat attribute, the overall threat assessment, and the categorization of high, moderate, and low for each.

These activities resulted in a threat assessment framework consisting of three threat attributes, each associated with three levels of assessment, and an overall assessment (Figure 2). The threat attributes were designed to assess the severity of harm to human health, the degree to which a threat is likely to impact Canada, and Canada’s capacity to prevent or mitigate the threat.

Figure 2 - Text description

The framework was designed as a qualitative tool to assess three distinct threat attributes (concern for human health, potential to affect Canada, and concern for insufficient control measures) that will be used to characterise a given public health threat that is anticipated to have an impact on Canada in some capacity. Each threat attribute is associated with high, moderate, and low assessment levels. The overall threat assessment is assigned based on the ratings for each of the threat attributes.

Once the framework was developed, a Delphi-based approach was used to define each threat attribute, the overall threat assessment, and the criteria for categorizing threats as high, moderate, and low. The Delphi technique is a scientific approach that engages experts in iterative surveys and group feedback to work towards consensus on challenging topics ^{Footnote 9}. Draft threat assessment definitions were developed by the research team to populate the framework. Consensus was defined, a priori, as 80% of participants in agreement with each definition, as worded.

Participant inclusion and recruitment

Invitations to participate in the expert panel were extended to all members of PHAC’s Scientific Committee for Coordinated Threat Assessment (SCCTA). The SCCTA is a group of subject matter experts with representatives from relevant disciplines within PHAC that meet weekly to assess public health threats. Participants were actively recruited to ensure representation from all stakeholder groups, reflecting the end-users of the framework for threat assessment from a national perspective.

Surveys

Agreement with definitions was assessed through three rounds of surveys conducted online using Microsoft Forms. Survey data were collected anonymously, and no directly identifying information was collected from participants. For the first two surveys, agreement with each definition was assessed using a five-point Likert scale with the following options: Yes, as currently worded; Yes, but change wording; Unsure; Probably not; and Definitely not. Respondents were also asked to provide detailed feedback or suggest alternative wording, which was used to inform modifications to the definitions.

Participants were given eight business days to complete each survey round. Two reminders were sent to participants for each survey. After each survey round, all participant feedback was considered in revising the definitions. Participants were then provided a summary of the results, reasons for changes to the definitions, and the next iteration of the definitions. Subsequent surveys and the consensus meeting included only items that had not reached 80% agreement. Following the initial consensus meeting, a third survey was conducted to formally assess agreement using a binary response format (i.e., yes/no).

Consensus meetings

Two consensus meetings were held by videoconference three weeks apart and were facilitated by a member of the research team. Results from both surveys and the modified definitions were presented at the first consensus meeting. Participants discussed each revised definition that had not reached consensus after the second survey. Areas of disagreement were explored, and definitions were further modified as needed. Consensus on each item discussed during the meeting was assessed using a post-meeting survey, with consensus defined as 80% agreement. A second consensus meeting was conducted to address outstanding areas of disagreement and to discuss the process by which the tool would be operationalized.

Data analysis

For each iteration of the survey, quantitative data were analyzed using simple descriptive statistics (i.e., the proportion of participants selecting each level of agreement) using R Studio ^{Footnote 10}. A qualitative thematic analysis ^{Footnote 11} was used to summarize free text data into broad categories, informing feedback to participants and revisions to the definitions.

Results

Participants

A total of 26 participants were included in the consensus process. Of the 10 PHAC program areas that regularly contribute to threat assessment, representatives from eight agreed to participate, along with additional members of the SCCTA, resulting in representation from 12 programs in total. Representatives included subject matter experts in zoonotic disease (n=1), outbreak management and food safety (n=2), respiratory virus epidemiology and surveillance (including COVID-19) (n=4), risk assessment (n=8), travel health (n=3), vaccine surveillance and readiness (n=2), emergency management (n=1), biosecurity (n=1), public health measures (n=2), risk sciences and modelling (n=1), and International Health Regulations (n=1).

Surveys

Of the 26 participants, there were 23 responses to survey 1 (88.5%), 19 responses to survey 2 (73.1%), and 10 responses to survey 3 (38.5%). Over the course of the study, there was a progressive, positive shift toward consensus with iterative modifications of the definitions (Table 1).

No definitions achieved consensus in the first survey, two achieved consensus in the second, and all definitions assessed after the consensus meetings reached 90% to 100% agreement, exceeding the 80% target. The definitions for high, moderate, and low overall threat assessment were not assessed in the final survey however, verbal agreement amongst attendees was achieved during the second consensus meeting. Thematic analysis of the written feedback resulted in categorization by the following broad themes: wording changes, framework language, and clarifications.

Consensus meetings

The first and second consensus meetings were attended by 15 (57.7%) and 17 (65.4%) participants, respectively, representing 11 of the 12 participating programs. Participants discussed and refined the definitions, raised methodological questions and program-specific issues about how to apply the resulting definitions, and commented on practical considerations for applying the tool (e.g., assessing international threats not yet present in Canada). These discussions prompted group exploration of related operational challenges. Participants informally reported a deeper understanding and greater level of comfort with the tool after engaging in these conversations. The final threat assessment definitions are presented in Table 2.

Discussion

The new threat assessment framework was designed to assess the severity of harm to human health, the degree to which a threat is likely to impact Canada, and Canada’s capacity to prevent or mitigate the threat. This article describes the Delphi consensus process used to determine a set of qualitative criteria with which users will assign a rating of high, moderate, or low for each of the three distinct threat attributes. Engaging a range of subject-matter experts led to consensus on 16 definitions in three distinct categories and one overall assessment category, resulting in a flexible framework capable of characterizing a wide range of public health threats.

The framework is aligned with other national and international health agencies in terms of the important elements to consider in public health threat assessment ^{Footnote 5Footnote 6Footnote 7} and qualitative flexibility ^{Footnote 5}. Moving from the previous algorithm-based approach to a qualitative framework allows for flexibility in weighting individual threat attributes according to their relative importance in the context of a specific threat. Unlike the former algorithm-based approach, which based assessments on federal actions, the new risk-based characterization provides a clearer representation of the key information used to assess a threat and can better inform appropriate public health actions. A separate publication provides an in-depth description of the threat assessment process at PHAC and includes an example of threat assessment in practice ^{Footnote 12}.

Several strengths were associated with using a Delphi-based approach to develop definitions for a threat assessment tool. Achieving consensus among experts from multiple subject-matter areas provides confidence that the process produced a robust tool capable of appropriately characterizing public health threats and that is applicable to multiple areas of public health. Including consensus meetings allowed participants to provide verbal feedback, which resulted in greater clarification of their views than was possible with surveys alone. This collaborative process fostered understanding and acceptance of the definitions across programs. Finally, programs expressed appreciation for the opportunity to participate in this process and contribute to the development of the tool.

Limitations

The process of developing the new threat assessment tool had several limitations. The initial steps to develop the framework were conducted within the research team and, while based on feedback from programs and an international scan, may have been designed differently if a process of systematic feedback had been used. Participation declined by 57% over the course of the Delphi study, and the anonymous nature of the surveys made it impossible to assess representativeness across PHAC programs. However, participation in the consensus meetings was largely representative of the complete list of participating programs, including programs that contribute regularly to the threat report. Finally, in terms of the Delphi methodology, the statistical stability of responses was not measurable because only two rounds of surveys were conducted ^{Footnote 13}.

Conclusion

Standardized, broadly accepted definitions for threat assessment in a federal public health setting are key to reliably characterizing and communicating public health events of importance and providing support for public health action. Ongoing work in this area will lead to more reliable and valid assessments of a variety of threats in multiple public health settings. Future plans include validating and evaluating the threat assessment tool, as well as continuing to apply it to threats beyond infectious diseases.

Authors' statement

The first author named is lead and corresponding author. All other authors are listed in alphabetical order. Co-author contributions are as follows:

GB — Conceptualization, methodology, investigation, formal analysis, project administration, writing–original draft, writing–review & editing
CD — Investigation, formal analysis, project administration, writing–review & editing
CE — Conceptualization, methodology, writing–review & editing, supervision
JM — Investigation, formal analysis, project administration, writing–review & editing
AS — Investigation, formal analysis, project administration, writing–review & editing
LW — Conceptualization, methodology, writing–review & editing, supervision

Competing interests

None.

ORCID numbers

None.

Acknowledgements

The authors gratefully acknowledge members of the Delphi panel for their engagement and expertise in the consensus process.

Funding

This work was supported by the Public Health Agency of Canada.

• Previous Page
• Table of contents
• Next page