Supplemental and Part B Questions

NOAA National Weather Service

Impact Based Warnings (IBW) Project

Part A and B DOC/NOAA Customer Survey Clearance

OMB Control No. 0648-0342

Impact Based Warnings Research Project

December 11, 2015

1. Supplemental Questions for DOC/NOAA Customer Survey Clearance
   (OMB Control Number 0648-0342)

1. Explain who will be conducting this survey. What program office will be
   conducting the survey? What services does this program provide? Who are the customers? How are these services provided to the customer?

NOAA’s National Weather Service (NWS) monitors, forecasts, and issues weather watches, warnings, and advisories for all weather types across all 50 states. In particular, the NWS local offices issue these warnings and other information products through websites, through NOAA Weather Radio, and through interactions with partners such as the media, government officials, emergency managers, and community groups. These warnings provide lifesaving information, such as when to shelter from a severe thunderstorm or tornado warning.

Examples of Impact Based Warning Tags

In the wake of the 2011 Joplin, MO, tornado—the deadliest to strike the United States since modern recordkeeping began in the 1950s—the NWS began considering ways to better alert the public to such dangerous weather event risks. The NWS Central Region conducted an experiment with five weather stations in Missouri and Kansas to test whether the incorporation of impact-based warning (IBW) information into tornado warnings would help people make quicker decisions. Along with new wording, threat tags were attached to the warnings based on the severity of the situation (see illustration below). In 2013, the experiment was extended to 38 weather offices throughout 12 additional states within the Central Region; this expansion continued in 2015 to include more offices in the Southern, Eastern, and Western Regions.

The NWS is looking for input on how the IBW threat tags are used by its partners, including NWS local office forecasters, broadcast meteorologists, and emergency managers, and how these tags subsequently impact, if at all, public response and decision making. In addition, the NWS is looking for input on alternative ways to convey levels of storm severity, if the threat tags are not meeting users’ needs. This evaluation will be useful in helping the NWS develop and evaluate its warning language to ensure it is meeting its mission to save lives and protect property. To accomplish this, the NWS seeks to conduct focus groups with emergency managers and broadcast meteorologists to discuss the IBW program and other ways to communicate levels of severity for convective storms.

2. Explain how this survey was developed. With whom did you consult regarding content during the development of this survey? Statistics? What suggestions did you get about improving the survey?

The NWS contracted with Eastern Research Group, Inc. (ERG) on the development of the focus group guide. ERG has significant experience in organizing and facilitating focus groups for federal agencies that focus on customer satisfaction and outcome attainment. To develop the focus group guide, ERG worked with Dr. Joseph Ripberger, Deputy Director for Research at the University of Oklahoma Center for Risk and Crisis Management. Dr. Ripberger has conducted many research projects related to risk communication and behavioral response, including work on studying public response to weather threat tags. ERG and Dr. Ripberger also worked closely with John Ferree, NWS Severe Storms Services Leader, Mike Hudson, Chief Operating Officer, National Weather Service Central Region, and Dr. Vankita Brown, NWS Social Scientist.

The focus group questions are divided into two sections. The first section focuses on new ideas to communicate levels of severity and urgency of action for thunderstorms and tornadoes, and subsequently how this information may impact decision processes. The second section focuses on the participants’ perceptions and use of the current IBW program.

3. Explain how the survey will be conducted. How will the customers be sampled (if fewer than all customers will be surveyed)? What percentage of customers asked to take the survey will respond? What actions are planned to increase the response rate? (Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to services provided by Web.)

The focus groups will be conducted in four locations chosen for their varying levels of severe weather experience: Raleigh, NC; Birmingham, AL; Norman, OK; and Des Moine, IA. ERG will contact the Warning Coordination Meteorologist (WCM) at each Warning Forecast Office in each of the locations to organize the focus groups. The WCMs will invite the emergency managers and media using their respective decision support list serve. . The emergency management customers could include local, state, regional, and even institutional (i.e., university, hospital, etc.) representatives. Broadcaster customers may include local affiliates (ABC, NBC, CBS and FOX). Depending on the use of radio in the area, National Public Radio or local radio broadcasters may also be included. In each location, we will aim to conduct three focus groups of no more than 15 participants per group (two with emergency managers and one with the media).

ERG will contact all of the WCMs to introduce the project, our role in the project, and request that focus groups be scheduled at a convenient time. The NWS local office will be responsible for inviting their media and emergency management partners. The NWS expects that 75 percent of the invitees will agree to participate in the focus groups with ERG, given the importance and level of interest in this project. We anticipate that travel or timing conflicts will be the prime reasons for invitees to decline participation. Dates and focus group sites will be chosen to maximize participation.

4. Describe how the results of this survey will be analyzed and used. If the customer population is sampled, what statistical techniques will be used to generalize the results to the entire customer population? Is this survey intended to measure a GPRA performance measure? (If so, please include an excerpt from the appropriate document.)

The NWS will use the information resulting from this data collection to help guide refinements in threat tag definitions and messaging to ensure the project is meeting its goal to facilitate improved public response. This information will also help the NWS determine if additional training is required for NWS local office forecasters to utilize the IBW program and/or if more partner coordination is necessary to make the program more successful.

ERG will perform thematic analyses on the detailed focus group notes to identify key findings and recommendations. The NWS is not using any statistical methods to select participants from the population and will select all population members.

The data do not directly contribute to a GPRA measure.

2. Collections of Information Employing Statistical Methods

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.

The potential respondent universe is based on the size of the emergency management community and media market for the chosen NWS offices, which varies from one location to another. Given busy schedules and the need for broadcast meteorologists/media to work around their broadcast air times, the total number of potential participants at each location is anticipated to be approximately 45 (30 emergency managers and 15 media) per location for a total of 180 participants in all four locations (see chart below).

Focus Group Sample

Location	Emergency Managers (Two groups per location)	Media (One group per location)	Totals
Raleigh, NC	30	15	45
Birmingham, AL	30	15	45
Norman, OK	30	15	45
Des Moines, IA	30	15	45
Totals	120	60	180

2. Describe the procedures for the collection, including: the statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Statistical Method for Stratification and Sample Selection

The NWS is not using statistical methods for collecting this data.

Estimation Procedure and Accuracy

The NWS does not need to extrapolate the results to the population and will therefore not need to estimate population parameters from the collected data.

Unusual Problems Requiring Specialized Sampling Procedures

None are required.

Periodic Data Collection Cycles

This request is for a one-time data collection.

3. Describe the methods used to maximize response rates and to deal with nonresponse. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield "reliable" data that can be generalized to the universe studied.

The NWS office in the region will send out the invitation to the focus groups. This invitation should help to encourage response since it will come from an entity the potential attendees will recognize. Additionally, the potential attendees have all previously worked closely with the NWS. The NWS will send out reminder invitations as well.

4. Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved OMB must give prior approval.

The NWS consulted with Eastern Research Group, Inc. (ERG) on the development of the focus group guide. ERG has significant experience assessing technical assistance provided by federal agencies through detailed interviews, focus groups, stakeholder engagement, and surveys that focus on customer satisfaction with services. The focus group guide development process was informed through discussions with NWS staff.

5. Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The NWS has contracted with Eastern Research Group, Inc. (ERG) of Lexington, MA, to design the focus group guide and implement the data collections. ERG’s project manager for this work is Gina Eosco (703-841-1705; [email protected]).

IBW Research Project 3