Significant Weather Event Communication Survey
for NOAA’s National Weather Service Hazard Simplification Project
At the American Meteorological Society’s Annual Meeting
November 12, 2014
Supplemental
Questions for DOC/NOAA Customer Survey Clearance
(OMB Control
Number 0648-0342)
Explain who will be conducting this survey. What program office
will be
conducting the survey? What services does this program
provide? Who are the customers? How are these services provided to
the customer?
The survey will be conducted by NOAA’s National Weather Service (NWS). The NWS forecasts hazardous weather situations and issues warnings, watches, advisories (WWA) and other information products to convey the threats posed by these events. These products are intended to help communities prepare for and respond to hazardous weather in order to protect people’s lives and property. The products are communicated to the public through websites, smart phones, television programs, radio broadcasts, and NOAA radio. NWS customers include weather professionals, transportation and aviation officials, emergency management personnel, public works departments, broadcast meteorologists and other media, and the public.
The NWS has embarked on an effort to simplify its WWA products, since both prior social science research and NWS service assessments have demonstrated that many members of the public, and even some NWS partners, don’t understand the distinctions among the terms used in the different WWA products or their intent. However, any change to the current WWA system must happen deliberately, gradually, and with transparency since the terms are recognized and widely used by NWS partners, as well as institutionalized into some kinds of societal decision-making.
This survey builds on and furthers social science research conducted in the summer of 2014 that involved focus groups with emergency managers, broadcast meteorologists, NWS Weather Forecast Office staff, and the public. The focus groups explored the current understanding and utility of the WWA system and possible enhancements to a new or modified system (ICR Reference Number 201103-0690-001, 3/14/14). This work indicated that there is a spectrum of understanding of the current WWA system and a difference of opinion on how much change is needed or desired to enhance the present system. It also showed considerable support for enhancing the current WWA system with simple explanatory language that could convey threats, impacts, and/or desired actions, although there was no agreement on what terms or phrases should be used. There was also considerable support for the use of a color scale to convey threat levels, with most participants favoring the red/orange/yellow spectrum. Opinions differed on whether a number scale or icons would be useful. This survey explores these potential modifications and other system parameters to get a better understanding of what changes or enhancements would be beneficial to make to the present WWA system.
Explain how this survey was developed. With whom did you consult regarding content during the development of this survey? Statistics? What suggestions did you get about improving the survey?
The NWS contracted with Eastern Research Group, Inc. (ERG) to develop the survey. ERG has significant experience assessing technical assistance provided by federal agencies through detailed interviews, focus groups, and surveys that focus on customer satisfaction and outcome attainment. To develop the interview and discussion questions, ERG worked with Dr. Gina Eosco, a well-known and respected social scientist in the weather community, who has extensively studied risk communication related to meteorological hazards. She received her PhD in Communications from Cornell. ERG also worked with Dr. Kim Klockow, a Post-Doctoral Fellow at the University Corporation for Atmospheric Research (UCAR), who is currently working with the NWS in the Office of Atmospheric Research and Office of Weather and Air Quality. She received her PhD in Geography from the University of Oklahoma. ERG also worked closely with NOAA and NWS leadership on the project, including Elliott Jacks, Chief, Fire and Public Weather Services Branch, and Jennifer Sprague, Acting Chief of Staff, National Ocean Services. Suggestions for improving the survey included shortening and simplifying the questions being asked and keeping the survey to less than 10 minutes, since it will be administered in an exhibit hall as part of a large conference with a fair amount of noise and other distractions.
The survey’s questions focus on three different WWA prototypes, one of which is the current system. The questions probe for respondents’ preferences for wording, color, and hazard depiction. The questions incorporate input on alternative system preferences from emergency managers, broadcast meteorologists, NWS Weather Forecast Office staff, and the public derived from focus groups conducted in the summer of 2014 (ICR Reference Number 201103-0690-001, 3/14/14).
These questions are designed to help the NWS understand what kinds of enhancements or changes to the current WWA system would be most beneficial.
Explain how the survey will be conducted. How will the customers be sampled (if fewer than all customers will be surveyed)? What percentage of customers asked to take the survey will respond? What actions are planned to increase the response rate? (Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to services provided by Web.)
The survey will be conducted at the American Meteorological Society’s (AMS’s) annual meeting in Phoenix, Arizona, on January 4-8, 2015. The AMS promotes the development and dissemination of information and education on the atmospheric and related oceanic and hydrologic sciences and the advancement of their professional applications. Founded in 1919, AMS has a membership of more than 14,000 professionals, students, and weather enthusiasts. AMS sponsors an annual meeting that is attended by approximately 5,000 weather professionals. A four-hour science and weather fair, called Weatherfest, will be held the day before the AMS meeting begins. Weatherfest is designed primarily for families and can attract up to 10,000 people.
The survey will mainly be targeting the weather professionals who attend AMS; it will also be taken by some members of the public who attend Weatherfest (i.e., those “invested” enough in weather information in the Phoenix area to attend the event).
The survey will feature three different prototypes as they would be displayed on www.weather.gov. It will include the current system that is currently seen on www.weather.gov. The prototypes will be displayed on laptops or tablets and will feature map views at national and regional levels, as is currently displayed on www.weather.gov.
AMS Meeting
The survey will be administered on laptops or tablets, and the booth will have eight stations available for respondents to take the survey. We anticipate that the booth will be staffed for 23 hours over a four-day period. Because the survey will take approximately 10 minutes to complete (including time to seat people and provide directions), we conservatively estimate that a maximum of 6 people will take the survey in an hour at any given station. There will be 8 stations; therefore, 48 people will be able to take the survey in an hour (6 people/hour x 8 stations).
Since the booth will be open 23 total hours, we anticipate that the maximum number of respondents will be 23 hours x 48 people = 1,104. Further, we anticipate that it is unlikely we’ll have 6 people every hour, so we conservatively estimate a maximum of 1,000 respondents or 20 percent of the 5,000 anticipated conference attendees will take the survey.
To increase survey response, several NWS staff and Dr. Eosco, all of whom will be speakers at the conference, will encourage members to visit the booth and take the survey as part of their presentations. Additionally, the NWS and Dr. Eosco will be part of a town hall meeting (expected attendance of 200 people) that will focus on the WWA hazard simplification project. The town hall will help raise awareness of the project and also encourage attendees to visit the NWS booth and take the survey.
Weatherfest
We will also conduct the survey at Weatherfest (for attendees who are 18 or older). Weatherfest is an event open to the public that offers education, outreach and activities about weather, water, and climate. The event is geared towards families, with every booth offering a hands-on activity for children. Because the booth space will also feature a children’s activity, we anticipate having 4 stations available in the booth for respondents to take the survey.
Because the survey will take approximately 10 minutes to complete (including time to seat people and provide directions), we conservatively estimate that a maximum of 6 people will take the survey in an hour at any given station. There will be 8 stations; therefore, 48 people will be able to take the survey in an hour (6 people/hour x 8 stations). Since the booth will be open 4 hours, we anticipate that the maximum number of respondents will be 36 hours x 4 people = 144 people.
To increase response rate, we will host an interactive children’s activity in the booth that will be designed to engage children while their parents take a few minutes to do the survey.
Describe how the results of this survey will be analyzed and used. If the customer population is sampled, what statistical techniques will be used to generalize the results to the entire customer population? Is this survey intended to measure a GPRA performance measure? (If so, please include an excerpt from the appropriate document.)
The data will be analyzed to identify trends among the respondents related to hurricane experience and concerns and to assess whether statistically significant differences exist between demographic and geographic groupings, with particular attention paid to potentially high-risk groups, both in terms of location and social vulnerability. The data gained from this survey will be reported to the NWS project leadership with the goal of further consider possible changes or modifications to the WWA system.
NWS will not be sampling from a population for this data collection. The data are intended to provide guidance from interested stakeholders in the way in which information should best be conveyed. Thus, no extrapolation to a population will be performed.
The data do not directly contribute to a GPRA measure.
Collections of Information Employing Statistical Methods
Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.
AMS Meeting
The potential respondent universe includes 5,000 AMS meeting attendees. The NWS estimates that time and space limitations will enable approximately 20 percent of these individuals, or 1,000 people, to take the survey. These individuals include federal, state, and local government employees, as well as private sector individuals (consultants, academics, nongovernmental organizations, broadcasters, media, and others). No statistical methods are being used in the participant selection.
Weatherfest
The potential respondent universe includes 10,000 public sector attendees. Time and space allows for approximately 144 individuals (age 18 and older) to be able to take the survey. No statistical methods are being used in the participant selection.
Describe the procedures for the collection, including: the statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.
Statistical Method for Stratification and Sample Selection
NWS is not using statistical methods for collecting these data.
Estimation Procedure and Accuracy
NWS does not need to extrapolate the results to the population and will therefore not need to estimate population parameters from the collected data. This also means that the accuracy of the estimates in not meaningful to calculate
Unusual Problems Requiring Specialized Sampling Procedures
None are required.
Periodic Data Collection Cycles
This request is for a one-time data collection.
Describe the methods used to maximize response rates and to deal with nonresponse. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield "reliable" data that can be generalized to the universe studied.
The survey will be conducted at stations in a booth at a conference. To increase survey response, several NWS staff and Dr. Eosco, whom will be speakers at the conference, will encourage members to visit the booth and take the survey as part of their presentations. Additionally, the NWS and Dr. Eosco will be part of a town hall meeting (expected attendance of 200 people) that will focus on the WWA hazard simplification project. The town hall will help raise awareness of the project and also encourage attendees to visit the NWS booth and take the survey.
Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved OMB must give prior approval.
The research gathered via the focus groups conducted under ICR Reference Number 201103-0690-001 (approved on 3/14/14) provided valuable input that has been used to develop this survey. The survey includes questions related to terms, colors, icons, and other system parameters that were suggested by attendees of those focus groups.
Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.
The NWS has contracted with Eastern Research Group, Inc. (ERG) of Lexington, MA, to design the survey instrument, develop the sampling approach, implement the survey, and analyze the resulting data collected. The survey design team included the following individuals:
Dr. Lou Nadeau (781) 674-7316; [email protected]
Linda Girardi (703) 841-0501; [email protected]
Dr. Gina Eosco (781) 704-4458;
Dr. Kim Klockow (301) 734-1068; [email protected]
Significant
Weather Event Communication Survey
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Lou |
| File Modified | 0000-00-00 |
| File Created | 2021-01-31 |