NWS Product Survey Supplemental and Part B questions

NOAA National Weather Service

Tropical Cyclone Products Testing

Part A and B DOC/NOAA Customer Survey Clearance

OMB Control No. 0648-0342

Expiration Date: 04/30/2015

NATIONAL WEATHER SERVICE (NWS) 2016 TROPICAL CYCLONE PRODUCTS TESTING

1. Supplemental Questions for DOC/NOAA Customer Survey Clearance
   (OMB Control Number 0648-0342)

1. Explain who will be conducting this survey. What program office will be
   conducting the survey? What services does this program provide? Who are the customers? How are these services provided to the customer?

NOAA’s National Weather Service (NWS) seeks to gather feedback via a Web-based survey from its core partners on two NWS forecast products associated with tropical cyclones:

1. The Hurricane Local Statement (HLS)/Tropical Cyclone Valid Time Event Code (TCV), an existing text product developed by local NWS Weather Forecast Offices (WFOs). The HLS/TCV is issued whenever a tropical cyclone is threatening the coast.

2. The Arrival of Tropical Storm Force Winds (ATSFW) graphic, a new map being created by the National Hurricane Center (NHC) that will depict the arrival of tropical storm force winds. By the time these winds arrive in an area, people should have finished their preparations and either be inside a safe shelter or left an area.

Both of these products will be posted on NWS websites when a tropical cyclone is threatening the coast and the NHC has issued a tropical cyclone public advisory.

The target audience for the testing includes broadcast meteorologists and emergency managers (EMs) who cover jurisdictions along the Atlantic Coast from Virginia to Texas, as well as well as Puerto Rico and the U.S. Virgin Islands. These partner groups depend upon and work closely with both the NHC and local WFOs whenever a tropical storm or hurricane threatens their area.

The survey is informed by the findings of four sets of focus group conducted on these products with these stakeholder groups in the summer of 2015 (ICR Reference #201504-0648-015).

2. Explain how this survey was developed. With whom did you consult regarding content during the development of this survey? Statistics? What suggestions did you get about improving the survey?

The NWS contracted with Eastern Research Group, Inc. (ERG) on the development of the survey. ERG has significant experience in conducting detailed interviews, focus groups, and surveys for federal agencies that focus on customer satisfaction and outcome attainment. To develop the survey, ERG worked with Dr. Betty Morrow, a well-known and respected social scientist, who has conducted many research projects with NWS stakeholder groups related to communicating and interpreting tropical cyclone hazards, as well as evacuation studies and risk behavior/communication studies. ERG and Dr. Morrow also worked closely with Dr. Rick Knabb, the director of the NHC, Robbie Berg, a forecaster with the NHC, Dr. Pablo Santos, the meteorologist-in-charge of the local Miami WFO, Dave Sharp the Science and Operations Officer for the Melbourne, Florida, WFO and Shannon White, with the NWS Advanced Weather Interactive Processing System (AWIPS) Support Branch. Suggestions for improving the survey focused on revising the map prototypes, rearranging the order of some questions, and adding opportunities for respondents to their feedback via a few, select open-ended questions.

3. Explain how the survey will be conducted. How will the customers be sampled (if fewer than all customers will be surveyed)? What percentage of customers asked to take the survey will respond? What actions are planned to increase the response rate? (Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to services provided by Web.)

The survey will be disseminated to directors of coastal state/county/parish EM agencies in Virginia, North Carolina, South Carolina, Georgia, Florida, Alabama, Mississippi, Louisiana, Texas, Puerto Rico, and the U.S. Virgin Islands. Potential respondents will be asked to participate via an email invitation. We will obtain an email contact list of the 11 state/territorial EM agencies listed above from the Federal Emergency Management Agency (FEMA), and then send a request to each state/territorial EM contact to request the email contact information for their coastal county contacts. Based on past surveys using the same methodology, such as the Survey of Coastal Managers Perspectives on NWS Storm Surge Information conducted by ERG, the National Center for Atmospheric Research (NCAR), and Dr. Morrow in 2012, contacting 11 state/territorial agencies and their local-level contacts yielded a potential response universe of 124 EMs. The prior survey resulted in 75 completed surveys, for a completion rate of over 60 percent (see Table 1 below).

Table 1. EM Responses for the 2012 Storm Surge Testing Project

Emergency Managers in Coastal States or Territories	Number Contacted	Number of Respondents
Alabama	2	0
Florida	36	23
Georgia	6	4
Louisiana	19	9
Mississippi	3	1
North Carolina	16	9
Puerto Rico	1	0
South Carolina	6	3
Texas	28	11
U.S. Virgin Islands	1	1
Virginia	6	4
Totals	124	75

For the media sample, the NWS will send an email invitation to chief broadcast meteorologists at the four major local television stations (ABC, CBS, Fox, NBC) servicing the coastal markets from Virginia to Texas. This approach is being chosen given that prior research indicates that most people rely on local television stations for their storm forecasts. A similar survey of broadcast meteorologists conducted for the NWS, Survey of Media Perspectives on NWS Storm Surge Information, conducted by ERG, NCAR, and Dr. Morrow in 2012, resulted in completions by 51 meteorologists, for a completion rate of about 42 percent.

The NWS plans to increase the response rate by sending a personal email invitation to potential respondents describing the importance of their participation. The NWS also will send several reminders to potential respondents to encourage them to fill out the survey.

4. Describe how the results of this survey will be analyzed and used. If the customer population is sampled, what statistical techniques will be used to generalize the results to the entire customer population? Is this survey intended to measure a GPRA performance measure? (If so, please include an excerpt from the appropriate document.)

NOAA/NWS will use the information resulting from this testing to help guide refinements to the HLS/TCV and the development of the ATSFW graphic. Ultimately, the data gained from this survey will contribute to the goal of improving the communication of hurricane forecasts, thus contributing to the NOAA goal of preserving life and property. The NWS is not using any statistical methods to select participants from the population, and the data do not directly contribute to a GPRA measure.

2. Collections of Information Employing Statistical Methods

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.

Potential respondents will be asked to participate via an email invitation. We will obtain state/territorial agency contacts from FEMA, and then ask these agencies for their local-level coastal contacts. We will use a list of media contacts from the NHC for the media sample. Based on past surveys using the same methodology (Survey of Coastal Managers Perspectives on NWS Storm Surge Information and Survey of Media Perspectives on NWS Storm Surge Information, both conducted by NCAR, ERG, and Dr. Morrow in 2012), the expected response rate for the collection is 51% (126 completed surveys out of a total sample universe of 245 participants). See Table 2 below for the breakdown of respondents by sector.

Table 2. Breakdown of Respondents by Sector and Expected Response Rate

	Federal	State	Local	Private	Total	Expected Response	Response Rate
Emergency Managers	0	2	122		124	75	60%
Media				121	121	51	42%
Total Sample	0	2	122	121	245	126	51%

2. Describe the procedures for the collection, including: the statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Statistical Method for Stratification and Sample Selection

The NWS is not using statistical methods for collecting these data.

Estimation Procedure and Accuracy

The NWS does not need to extrapolate the results to the population and will therefore not need to estimate population parameters from the collected data. This also means that the accuracy of the estimates in not meaningful to calculate.

Unusual Problems Requiring Specialized Sampling Procedures

None are required.

Periodic Data Collection Cycles

This request is for a one-time data collection.

3. Describe the methods used to maximize response rates and to deal with nonresponse. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield "reliable" data that can be generalized to the universe studied.

The NWS plans to increase the response rate by sending a personal email invitation to potential respondents describing the importance of their participation. The NWS also will send several reminders to potential respondents to encourage them to fill out the survey.

4. Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved OMB must give prior approval.

The survey will be pre-tested with two broadcast meteorologists and two emergency managers. They will be sent a survey link for completing the survey (which will be formatted in a software package such as Qualtrics). We will ask these respondents how long it took to complete the survey as well as for any feedback improving the survey, such as refining any questions or instructions that were unclear or hard to follow.

5. Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The NWS has contracted with Eastern Research Group, Inc. (ERG) of Lexington, MA to design the interview guide and discussion questions and implement the data collections. ERG’s project manager for this work is Linda Girardi (703-841-0501; [email protected]).

5