Fire Weather Survey supplemental and Part B questions

Public Survey

for NOAA’s Fire Weather Risk Messages

December 18, 2017

1. Supplemental Questions for DOC/NOAA Customer Survey Clearance
   (OMB Control Number 0648-0342)

1. Explain who will be conducting this survey. What program office will be
   conducting the survey? What services does this program provide? Who are the customers? How are these services provided to the customer?

This request is for a public survey to be conducted by NOAA’s National Weather Service (NWS) to assess how best to communicate fire weather risk information. The surveys are similar to a set of surveys approved by OMB under this control number on 11/20/17 (Public Survey for NOAA’s National Weather Service Hazard Simplification Project). Those surveys covered a number of hazards, but did not include fire weather. Furthermore, the survey being described in this submission is part of a separate effort at the NWS. The NWS has designed this survey to be consistent with the surveys that were previously approved to build a stronger base of information to assess potential changes to the NWS messaging overall.

The NWS currently forecasts fire weather situations and issues “Fire Weather Watches” and “Red Flag Warnings” in response to forecasted conditions in wind, humidity, and moisture content of fuels (e.g., grass). Historically, the NWS has targeted watches and warnings only to public officials such as emergency planners, first responders, and firemen. These terms (Fire Weather Watch and Red Flag Warning), however, have also been disseminated to the public through the media, websites, and social media—and this dissemination has expanded in recent years along with people’s use of the internet. Thus, the NWS is interested in knowing how well these terms are understood, whether they need to be changed, and the extent to which changes would be welcomed by the public. Presumably, a better-informed public in the face of wildfires should result in fewer fatalities and injuries associated with wildfires.

This survey builds on a set of focus groups that was approved under this control number on 3/13/17 (Public Focus Groups to Assess Fire Weather Services). The report presented to NWS by its subcontractor (Eastern Research Group, Inc. (ERG)) had three main findings:

• Is the phrase “Red Flag Warning” intuitive to the public? The focus group results indicated that although the phrase is very familiar to residents in western United States who frequently experience wildfires, it is not intuitive in and of itself. Without outreach and continual education, the term does not convey wildfire risks to the public.

• Is there alternative messaging that would better convey wildfire risks to the public? Based on the focus group feedback, short, urgent messaging that expresses levels of wildfire severity (e.g., through a color scale or heightened language) and more explicitly states the weather hazard would better convey wildfire risks to the public, particularly for visitors to wildfire-prone locations that come from locations that have little to no experience with wildfires. Conveying levels of severity would also help the public better prepare and avoid complacency when they experience Red Flag Warnings multiple days in a row. The focus group participants also thought that the NWS should reconsider its Fire Weather Watch, because its definition is too close to the Red Flag Warning and it does not adequately convey a lower level of severity.

• How can the NWS and partners best disseminate fire weather messaging to the public? Focus group participants felt that a combination of public outreach and education by local weather forecast offices (WFOs), partnerships with other federal agencies and private-sector entities, highly visible and up-to-date signage on roadways and in parks, and Wireless Emergency Alerts (WEAs) would all be necessary to effectively disseminate fire weather messaging to the public.

The survey being submitted in this request expands on the focus group findings and tests out new messages that were derived from the discussion in the focus groups.

NWS has created three “prototypes” for new ways of communicating fire weather information using different words and/or colors. Although these prototypes were developed with input from public focus groups, the focus groups could be considered a small and unrepresentative sample of the public potentially affected by fire weather. The survey being submitted under this request will expand to a larger sample and will include groups that were not targeted in the focus groups.

The current system consists of two levels of conveying risk: a Fire Weather Watch and a Red Flag Warning. These two current message levels correspond to situations where conditions are developing that could lead to fires. For the survey, the NWS developed three new prototypes that contain corresponding “watch” and “warning” level messages, but that also add two new levels to these messages: an “observed fire” level and an “emergency” level. The NWS added the “observed” and “emergency” levels because participants in the focus groups indicated that some messages should correspond to situations where fire is “on the ground.” The four message sets (current system and three prototypes) and their corresponding levels appear in Table 1.

Table 1: Prototypes and Their Associated Levels

Prompt Level	Current	Prototype A	Prototype B	Prototype C
Watch	Fire Weather Watch	Fire Weather Alert	Moderate Fire Weather Warning	Level Yellow Fire Weather Warning
Warning	Red Flag Warning	Fire Weather Watch	High Fire Weather Warning	Level Orange Fire Weather Warning
Observed	NA [a]	Wildfire Warning	Severe Wildfire Warning	Level Red Wildfire Warning
Emergency	NA [a]	Wildfire Emergency	Fire Storm Warning	Level Purple Wildfire Warning

[a] The current system does not have prompts at these levels and thus will not be included in the survey.

The levels can be described as follows:¹

• Watch – “…upcoming weather conditions could result in critical fire weather conditions, which are expected to develop in the next 12 to 48 hours, but not more than 72 hours. In cases of dry lightning, a Fire Weather Watch may be issued for the next 12 hours.”

• Warning – “NWS issues a Red Flag Warning … to alert land managers to an ongoing or imminent critical fire weather pattern. NWS issues a Red Flag Warning when fire conditions are ongoing or expected to occur within the next 24 hours.”

• Observed – Fires are currently occurring on the ground, but the weather conditions are not currently driving the fire toward populated/ residential areas.

• Emergency – This level should imply a wildfire that is raging out of control. If is often hard to predict these fires because such they behave erratically, sometimes dangerously.

The prototypes can be described as follows:

• Current system – This prototype is the current WWA system.

• Prototype A – This prototype begins with fire weather conditions and progresses to actual fire terminology.

• Prototype B – This prototype uses adjectives to convey levels of risk.

• Prototype C – This prototype uses a color scheme instead of risk-based wording.

A key aspect of the surveys will be to test the prototypes using changes in severity over time (i.e., upgrading the risk). For example, as conditions related to fire weather develop, the NWS currently issues a “Fire Weather Watch” which may then be later upgraded to a “Red Flag Warning” as the forecasted conditions worsen. The NWS has developed two scenarios for this survey:

• Scenario 1 - A warning that is upgraded to an emergency (i.e., the occurrence of a wildfire). This scenario will involve respondents seeing (1) a watch-level message, (2) a warning-level message, (3) an observed fire-level message, and (4) an emergency-level message.

• Scenario 2 - A warning that is repeated (i.e., no wildfire develops). This scenario will involve respondents seeing (1) a watch-level message, (2) a warning-level message, and (3) a repeat of the warning-level message.

The NWS determined that these two scenarios reflected likely cases in the ways in which the public may experience fire weather messages. Additionally, prior to receiving the watch-level prompt in both scenarios, respondents will see a generic “base information” prompt that includes none of the language being tested, but sets a baseline level of response for each respondent.

2. Explain how this survey was developed. With whom did you consult regarding content during the development of this survey? Statistics? What suggestions did you get about improving the survey?

The NWS contracted with ERG to develop the surveys. ERG has significant experience assessing technical assistance provided by federal agencies through detailed interviews, focus groups, and surveys that focus on customer satisfaction and outcome attainment.

As noted, the surveys focus on four (current approach and three alternatives) different message types. As noted, the prototypes evolved from input gathered from public focus groups as well as from interviews with emergency managers and other fire weather stakeholders.

ERG used statistical power analysis to verify that the sample size being used would allow for meaningful comparisons given the survey design. As will be discussed in Part B below, the sample for this set of surveys was set at approximately 900 total respondents. Based on ERG’s analysis (described in Part B), the number of respondents for each hazard will allow for identifying reasonable differences between reactions to the prototypes at sufficient statistical power that will the NWS make informed decisions on how to proceed with modifying future messages. ERG’s statistical power calculations are detailed in Part B.

3. Explain how the survey will be conducted. How will the customers be sampled (if fewer than all customers will be surveyed)? What percentage of customers asked to take the survey will respond? What actions are planned to increase the response rate? (Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to services provided by Web.)

How the Survey Will Be Conducted

The NWS will perform an online survey to collect these data since the NWS distributes much of its information though online web-based sources, including fire weather information. Additionally, online sources, especially mobile sources, are particularly important in conveying risks associated with fast-developing events such as wildfires. The NWS’s contractor, ERG, will work with Qualtrics, Inc., a leading provider of online survey services. ERG will instruct Qualtrics to select random samples from a pre-determined set of geographic areas (described in Part B below).

Response Rate

ERG expects that 70 percent of respondents will take the survey. This rate is based on ERG’s prior work implementing similar surveys for the NWS.

Maximizing Response

To ensure a maximum response rate, the NWS’s contractor ERG will perform the following tasks:

• ERG has developed a survey that minimizes the burden on respondents by using good survey design. This includes developing well-written questions and limiting the number of the questions to the minimum necessary.

• ERG will require Qualtrics, Inc. to select respondents from a pool of individuals who have indicated they are willing to take surveys. Many email survey lists are constructed from individuals who passively opt in to taking surveys (e.g., by agreeing to a terms of service agreement on a web site). ERG will require Qualtrics to use survey lists where those on the lists consciously indicated they would be willing to take surveys (i.e., passive opt-ins will be excluded).

• ERG will use multiple prompts to generate responses. ERG will use a pre-notification email to respondents, an email that asks the respondent to take the survey, and then two reminder emails.

In addition: a graduate student researcher at the University of Georgia (Castle Williams) developed a survey that assesses the NWS’s hazard messages for extreme heat. As part of his research, Mr. Williams has performed a series of cognitive tests with potential respondents to improve his survey design. He also implemented a pre-test of his survey instrument. Mr. Williams’ survey is similar to the one being implemented here and shares a number of similar question types. Mr. Williams shared his initial results with NWS and ERG, which allowed ERG to incorporate results into the design of this survey.

4. Describe how the results of this survey will be analyzed and used. If the customer population is sampled, what statistical techniques will be used to generalize the results to the entire customer population? Is this survey intended to measure a GPRA performance measure? (If so, please include an excerpt from the appropriate document.)

The survey data will be analyzed by comparing respondent preferences for different prototypes. As noted, the purpose of the data collection effort will be to assess how the public, a primary NWS customer, reacts to different warning levels and terminology for fire weather. ERG will use the survey data to determine which prototypes are more preferred among potential customers by comparing the percentages of NWS customers who prefer which prototype. These results will assist NWS leadership with considering possible changes or modifications to the WWA system.

ERG will use sample weights to generalize the survey results to the populations.

The data do not directly contribute to a GPRA measure.

2. Collections of Information Employing Statistical Methods

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.

Respondent Universe

The NWS conducted focus groups in a set of western states (New Mexico, Wyoming, Colorado, Montana, and California) due to the increased prevalence of wildfires in those states. However, focus group participants indicated that visitors and second homeowners from areas that experience wildfires less frequently should also be a target of the NWS’s messages. Thus, the NWS has determined that samples should be selected for two sets of states:

• States that frequently experience wildfires (focus group states). This would include the states where the NWS performed focus groups (New Mexico, Wyoming, Colorado, Montana, and California). The NWS will target a total of 600 total respondents across these five states with no more than 25 percent of the total sample (150 respondents) coming from either California and Colorado, the two most populous states among the five. Furthermore, the NWS will ensure that at least 75 percent of the respondents come from “non-urban” locations.

• States that are less experienced with wildfires. The NWS will select a sample of 300 respondents from Alabama, Georgia, North Carolina, Pennsylvania, and Virginia; these states each experience some risk of wildfires. Additionally, residents from these states may also take trips to the West and thus experience wildfire risks during those trips. As with the focus group states, the NWS will select a sample comprise of at least 75 percent of respondents from “non-urban” locations.

Table 2 summarizes the geographic focuses described above, the adult population (age 20 and older) for each selected area, the sample that will be selected, the anticipated response rate, and the targeted sample size.

Table 2: Geographic Focus, Populations, and Sample Information

Set of States	Population Over the Age of 18 [a]	Non-Urban Population Over 18 [b]	Sample [c]	Response Rate	Targeted Sample [d]
Focus group states (CA, CO, MT, NM, WY)	35,339,000	5,502,000	860	70%	600
States with less experience with wildfires (AL, GA, NC, PA, VA)	16,406,000	3,879,000	430	70%	300

[a] U.S. Census Bureau estimates taken from 2010 Census. Values were rounded to the nearest thousand.

[b] Reflects adult population that lives in “rural” areas or in “urban clusters” (not in “urbanized areas”) as defined by the Census Bureau. The NWS expects people who live in “rural” areas or in “urban clusters” best reflect the areas that are at most risk from wildfires.

[c] Calculated by dividing the target sample size by the response rate and rounding to the nearest ten.

[d] Sample size calculations are provided below in Section B, Question 2.

Selection Method

The NWS will use random selection for the sampling. ERG, the NWS’s contractor, will instruct Qualtrics (the company that will maintain the sampling list) to randomly select the numbers listed in Table 2 above. ERG expects that 70 percent of those will respond to the survey. In the event that fewer respond or some initial selections are out of scope (e.g., not an adult), ERG will instruct Qualtrics to select replacements and to target the sample size listed in Table 2.

2. Describe the procedures for the collection, including: the statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Stratification, Sample Size, and Precision and Accuracy

The sample size for these surveys was primarily determined by available budget. In winning the contract to perform the work, ERG proposed a sample of approximately 750 respondents; the sample size, however, was based on the assumption that only western states would be part of the survey effort. ERG increased the sample to 900 to account for the inclusion of non-western states in the sample based on the results of the focus groups.

The survey will involve comparing responses between groups that see different types of messages (the current system plus three prototypes); thus, the number who see the different prototypes will be a key factor in determining precision and statistical power. Additionally, the NWS has incorporated two scenarios to present the prototype in the survey design: (1) warning upgraded to an emergency and (2) a warning repeated. Thus, with four message sets (the current system plus three prototypes) and two scenarios, there are eight distinct groupings (4 x 2 = 8). Both sets of states will see the same survey questions, so each group has approximately 112 respondents (900 ÷ 8).²

The NWS calculated the statistical power to detect differences between the comparison groups assuming that a 5-point scale question would be used for the comparison.³ Statistical power reflects the probability of detecting a difference between two samples assuming that the difference exists among the populations. The NWS calculated the statistical power of detecting a difference (precision) of 0.25 points, 0.5 points, and 0.75 points in the means of a 5-point scaled question between two sub-samples. The NWS also assumed three scenarios for the standard deviation of the responses: a worst-case scenario corresponding to a standard deviation of 2.0⁴ and two more realistic scenarios with standard deviations of 1.5 and 1.0. The statistical power calculations for the three levels of precision and the three assumed standard deviations, using a sample of 112 per group, appears in Table 3. Tests with “strong” statistical power tend to be close to 75 percent or higher. The samples sizes for this survey will provide strong power for detecting 0.75-point differences at all assumed standard deviations and a 0.5-point differences at assumed standard deviations of 1.5 and lower. The NWS has assumed that the worst-case scenario for standard deviation is unrealistic and that 0.5-point changes in a mean value would be relevant; thus, the NWS is confident that the samples will allow for strong tests to compare respondent preferences and understanding between the prototypes.

Table 3: Statistical Power for Different Levels of Precision and Assumed Standard Deviations

Precision (difference in mean)	S.d.= 2	S.d. = 1.5	S.d. = 1
0.25	24%	35%	59%
0.5	59%	80%	98%
0.75	88%	98%	~100%

Note: “S.d.” refers to standard deviation.

Unusual Problems Requiring Specialized Sampling Procedures

None are required.

Periodic Data Collection Cycles

This request is for a one-time data collection.

3. Describe the methods used to maximize response rates and to deal with nonresponse. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield "reliable" data that can be generalized to the universe studied.

Maximizing Response Rates

The NWS’s contractor, ERG, will employ the following strategies to maximize response rates from the sample:

• ERG has developed a survey that minimizes the burden on respondents by using good survey design. This includes developing well-written questions and limiting the number of the questions to the minimum necessary.

• ERG will require Qualtrics, Inc. to select respondents from individuals who indicated they are willing to take surveys. Many email survey lists are constructed from individuals who passively opt in to taking surveys (e.g., by agreeing to a terms of service agreement on a web site). ERG will require Qualtrics to use survey lists where those on the lists consciously indicated they would be willing to take surveys (i.e., excluded passive opt-ins).

• ERG will use multiple prompts to generate responses. ERG will send a pre-notification email to respondents about the survey, an email that asks the respondent to take the survey, and then two reminder emails.

• ERG is implementing the survey in geographic areas where wildfires are a relevant risk. Furthermore, the recent wildfires in California will create a sense of importance to respondents on taking this survey.

Adequacy for Intended Uses

The purpose of these surveys is to get feedback on possible new approaches to presenting hazard warning risk information for wildfires and to assess how well the information would work among a new targeted audience (the public). The key information that the NWS needs are the public’s understanding of the new prototypes and the public’s satisfaction with the new prototypes. The NWS will use the information from these surveys in conjunction with other data to consider potential changes in fire weather warnings. No decisions are being made solely from the data being collected from these surveys. As such, the NWS expects the attainable precision (described above) is more than adequate given the purpose of these data. This is based on several considerations:

• The surveys involve collecting data from a large number of respondents across the affected areas (900 in total). Thus, the NWS will have a large number of data points to use in assessing the prototypes.

• The surveys are being implemented in a way that mimics how the NWS warns the public of fire risks; that is, the NWS issues a series of watches and/or warnings and then upgrades those over time as conditions worsen. Understanding how people respond to those upgrades (or a lack of an upgrade) is important.

• The prototypes are “directional” in nature rather than specific. The NWS has developed the prototypes based on input from focus groups and other prior research. The NWS also had meetings with internal staff who are responsible for fire weather hazard warnings. This survey will provide information on the types of messages that will work better with the public compared to others, but not necessarily the specific messages to use.

• Identifying large differences in public preferences for prototypes is acceptable. As noted above, we can expect to find 0.5-point differences in mean values on a 5-point scale between sub-groups in our sample with high statistical power. A 0.5-point difference would indicate a very large preference for one prototype over another; however, based on the calculations summarized in Table 3, and assuming a reasonable amount of variation in the data, the statistical power of finding mean differences of 0.5 (or even 0.25) point is high. A 0.5-point difference on a 5-point scale in mean preference between two prototypes is acceptable to the NWS.

4. Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved OMB must give prior approval.

The NWS’s contractor, ERG, has or will perform the following procedures and tests to ensure an effective data collection process:

• ERG has performed significant prior research on this subject as described earlier. ERG performed a series of group interviews with NWS staff and stakeholders as well as a series of focus groups with the public on the NWS’s dissemination of fire weather information. The results from these interviews and focus groups have been incorporated into the survey.

• This survey is modeled after a Hazard Simplification survey that OMB approved on 11/20/17 under this same control number. Thus, this survey incorporates the information we learned from developing that survey into this one as well. Furthermore, that survey effort involved an internal validation done by ERG staff and a pre-test of a similar instrument by a University of Georgia graduate student. Thus, all of that prior work was used as a basis for constructing this instrument.

• The pretest will consist of assessing results of the first day of the survey. ERG will make changes as needed based on the pre-test.

5. Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The NWS has contracted with Eastern Research Group, Inc. (ERG) of Lexington, MA, to design the survey instrument, develop the sampling approach, implement the survey, and analyze the resulting data collected. The survey design team included the following individuals:

Dr. Lou Nadeau (781) 674-7316; [email protected]

1 Quoted text in the bullets is taken from: [a] See http://www.nws.noaa.gov/om/fire/ww.shtml.

2 NWS is interested in differences between the two sets of states and will perform analyses to assess those differences. The primary purpose of this survey, however, is to assess differences between the current system and the three new prototypes.

3 The 5-point scale question is respondent preferences for a prototype or their understanding of the information.

4 The worst-case standard deviation for a 5-point scaled question corresponds to one half of the respondents selecting the lowest level (1) and one half selecting the highest level (5).

Fire Weather Communication Survey 10