Supplemental and Part B Questions

NOAA National Weather Service

Arrival of Tropical Storm Force Winds Social Science Project

Part A and B DOC/NOAA Customer Survey Clearance

OMB Control No. 0648-0342

Expiration Date: 04/30/2015

Arrival of Tropical Storm Force Winds Social Science Research Project

January 8, 2015

1. Supplemental Questions for DOC/NOAA Customer Survey Clearance
   (OMB Control Number 0648-0342)

1. Explain who will be conducting this survey. What program office will be
   conducting the survey? What services does this program provide? Who are the customers? How are these services provided to the customer?

NOAA’s National Weather Service (NWS) and National Hurricane Center (NHC) predict tropical cyclone storm movement and impacts and issues warnings, watches, and other information products. These products and services are provided through NWS Web sites, NOAA Weather Radio, and through interactions with partners such as the media, government officials, and emergency managers.

N WS Service Assessments, such as those for Hurricane Irene and Sandy, identified a number of gaps in the way NOAA articulates the risk from tropical cyclones in its various products, information, and services—at both the national and regional levels. Additionally, NWS partners have expressed a strong desire to have NHC create a separate product depicting the potential time of arrival of sustained winds of tropical storm force. Such a product will enable emergency managers and other officials to make more informed decisions on when to conduct and complete preparations as a tropical cyclone approaches. Broadcast meteorologists also have a need for this information for use in their on-air programming, websites, and social media.

The NWS is looking for input on what its partners need—and how they would use—a graphic and data depicting the arrival of tropical storm force winds. The primary customers of this new product are broadcast meteorologists and emergency managers. (A prototype of the map is shown at left.)

The NWS wants to ensure smooth development and implementation of this new product and is therefore seeking approval to collect information on how these partners currently determine the arrival of tropical storm force winds, as well as what their requirements are from NHC for datasets, graphic displays, and level of confidence. NWS is also interested in learning how the product could be interpreted in different locations and situations. NWS has draft prototypes of the graphic for partners to assess, but is also interested in learning if there are additional ways to present this information.

To accomplish this, NWS seeks to conduct one-on-one webinars with a small group of emergency managers and broadcast meteorologists. These interviews will be with a mix of both experienced and less experienced individuals in both coastal and inland areas, as well as those areas that have large populations to evacuate.

2. Explain how this survey was developed. With whom did you consult regarding content during the development of this survey? Statistics? What suggestions did you get about improving the survey?

NWS contracted with Eastern Research Group, Inc. (ERG) on the development of the interview guide and questions. ERG has significant experience in conducting detailed interviews, focus groups, and surveys for federal agencies that focus on customer satisfaction and outcome attainment. To develop the interview guide and discussion questions, ERG worked with Dr. Betty Morrow, a well-known and respected social scientist, who has conducted many research projects with NWS stakeholder groups related to communicating and interpreting tropical cyclone hazards, as well as evacuation studies and risk behavior/communication studies. ERG and Dr. Morrow also worked closely with Dr. Rick Knabb, the Director of the NHC, and Robert Berg, an experienced NHC forecaster.

The interview questions for emergency managers focus on how they currently determine and use arrival of tropical storm force wind information, particularly in their use of a computer program called HURREVAC, since any product NHC develops will have to be compatible with this system. HURREVAC is a privately-developed software program used by government emergency managers to track and analyze the threat for hurricanes. The HURREVAC program is supported by the Federal Emergency Management Agency, the U.S. Army Corps of Engineers, and NOAA.

Questions for broadcast meteorologists focus how they currently use and talk about this information, as well as how NHC should deliver the data so it can be used on the air.The questions are designed to help NWS understand what type of product to build and how it should be delivered, as well as what kinds of training or partner coordination will be necessary to use the new product effectively (along with other resource and tools the users may already be effectively using).

Suggestions for improving the interview guide and discussion group questions included adding a section about eliciting feedback on how to build confidence into the product and asking the interviewees if it would be useful to build more than one confidence level (such as 10 percent and 50 percent exceedance levels) into the product.

3. Explain how the survey will be conducted. How will the customers be sampled (if fewer than all customers will be surveyed)? What percentage of customers asked to take the survey will respond? What actions are planned to increase the response rate? (Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to services provided by Web.)

The survey will be in the form of one-on-one telephone interviews in a webinar format. Interviewees will be able to view draft prototypes of the graphic on their computer through a Go-To-Meeting type of format. NWS will have ERG and Dr. Morrow schedule and conduct an interview with up to 8 emergency managers and up to 8 broadcast meteorologists. The interviewees will be selected by NHC based on its knowledge of the partner community. The interviewees from the emergency management community will represent both local and state government agencies in inland and coastal areas. The interviewees with broadcast meteorologists will include individuals from private entities such as The Weather Channel and CNN, as well as those with local stations in inland and coastal locations.

ERG will contact all of interviewees to introduce the project, explain our role in the project, and to schedule an interview at a convenient time for the interviewees. NWS expects that 75 percent of the interviewees will schedule an interview with ERG, given the importance and level of interest in this product. NWS/NHC will provide additional candidates as alternatives for any individuals who decline to participate in the interview.

In scheduling each interview, ERG will introduce ourselves, explain our affiliation, describe what information we are seeking, and how long the interview will take:

My name is XX. I am with Eastern Research Group (ERG), and I am calling to schedule an interview with you on behalf of NOAA’s National Weather Service.  I am hoping to ask you some questions about a new product that the National Weather Service is looking to create that would provide information on the arrival of tropical storm force winds. The interview would be 40 to 60 minutes long.  Do you have any availability over the next couple of weeks to schedule this interview? We will not release your name or information that could identify you as part of this interview process or in our subsequent reports to the National Weather Service.

ERG will generate detailed notes from the interviews.

4. Describe how the results of this survey will be analyzed and used. If the customer population is sampled, what statistical techniques will be used to generalize the results to the entire customer population? Is this survey intended to measure a GPRA performance measure? (If so, please include an excerpt from the appropriate document.)

NWS will use the information resulting from this data collection to help guide the development and refinement of the arrival of tropical storm force winds product. This information will help NWS/NHC choose colors, legends, and labels for the graphic. It will also help NWS/NHC determine what level of confidence to build into the product and how to ensure compatibility with HURREVAC and the vendors who develop graphics for television broadcasts. Finally, this information will help guide NWS/NHC in determining what training and coordination efforts may be needed to ensure the products smooth implementation.

NWS will have ERG perform thematic analyses on the detailed interview notes to identify key findings, barriers, and challenges to developing the arrival of tropical storm force winds product.

The data do not directly contribute to a GPRA measure.

NWS is not using any statistical methods to select participants from the population and will select all population members in the discussion group and interviews.

2. Collections of Information Employing Statistical Methods

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.

The potential respondent universe includes 16 individuals (8 emergency managers and 8 broadcast meteorologists). No statistical methods are being used in the participant selection.

2. Describe the procedures for the collection, including: the statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Interviews with Broadcast Meteorologists and Emergency Managers

Statistical Method for Stratification and Sample Selection

NWS is not using statistical methods for collecting these data.

Estimation Procedure and Accuracy

NWS does not need to extrapolate the results to the population and will therefore not need to estimate population parameters from the collected data. This also means that the accuracy of the estimates in not meaningful to calculate

Unusual Problems Requiring Specialized Sampling Procedures

None are required.

Periodic Data Collection Cycles

This request is for a one-time data collection.

Periodic Data Collection Cycles

This request is for a one-time data collection.

3. Describe the methods used to maximize response rates and to deal with nonresponse. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield "reliable" data that can be generalized to the universe studied.

ERG will contact all of interviewees to introduce the project, our role in the project, and to schedule an interview at a convenient time for the interviewees. NWS expects that 75 percent of the interviewees will schedule an interview with ERG given the importance and level of interest in this product. NWS/NHC will provide additional candidates for any individuals who decline to participate in the interview.

4. Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved OMB must give prior approval.

NWS consulted with Eastern Research Group, Inc. (ERG) on the development of the interview guide and discussion group questions. ERG has significant experience assessing technical assistance provided by federal agencies through detailed interviews, focus groups, stakeholder engagement, and surveys that focus on customer satisfaction with services. The interview guide and question development process was informed through discussions with NHC staff .

5. Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

NWS has contracted with Eastern Research Group, Inc (ERG) of Lexington, MA to design the interview guide and discussion questions and implement the data collections. ERG’s project manager for this work is Linda Girardi (703-841-0501; [email protected]).

Social Science Project on the Arrival of Tropical Storm Force Winds Graphic 4