Supplementary- Revised

HURRICANE FORECAST IMPROVEMENT PROGRAM (HFIP) SOCIO-ECONOMIC RESEARCH PROJECT

Part A: Supplemental Questions for DOC/NOAA Customer Survey Clearance (OMB Control No. 0648-0342)

1. Explain who will be conducting this survey. What program office will be
   conducting the survey? What services does this program provide? Who are the customers? How are these services provided to the customer?

This survey is being conducted by the National Weather Service’s (NWS’s) Hurricane Forecast Improvement Program (HFIP) to obtain feedback from residents of hurricane-affected areas about the usefulness of a number of new Web-based storm surge and wind graphics. A social science component of HFIP is studying how hurricane forecast communication might be improved. As part of that endeavor, NWS is planning to conduct the survey being described here. The purpose of this Web-based survey is to better understand how the public will be served by the proposed new forecast communication graphics and strategies. The survey targets individuals who are computer-literate and are familiar with NWS products. In addition to providing feedback, the respondents will benefit from having to consider hurricane hazards in responding to the survey.

2. Explain how this survey was developed. With whom did you consult during the development of this survey on content? Statistics? What suggestions did you get about improving the survey?

The questions and graphics that are a critical part of this survey evolved over a six-month period of informal discussions that Dr. Betty Morrow, consultant to the Eastern Research Group (ERG), NOAA’s subcontractor for this effort, had with citizens, emergency managers, and NWS forecasters. These discussions were conducted through one-on-one webinars of less than nine individuals or informally at conferences. The prototypes were adjusted throughout this process. A poll of the resulting graphics was conducted at an exhibitors’ booth sponsored by the National Hurricane Center at the 2012 National Hurricane Conference (OMB Control No. 0690-0030). During this entire informal discussion period, different stakeholders provided commentary and informal assessments of the graphics. As a result of this input, changes have been made in the choice of colors, backgrounds, placement of legends and methods of data presentation.

A list of suggested topics to be included in the proposed survey was circulated among the project leadership. This includes Jamie Rhome and Robert Berg at the National Hurricane Center, Jennifer Sprague and Jesse Feyen at NOAA headquarters, Linda Girardi at Eastern Research Group, Inc. (ERG), NOAA’s subcontractor, and Dr. Betty Morrow. The questions were then drafted by Dr. Morrow, using successful survey questions from past surveys where available and constructing new ones where needed. This draft set of questions was circulated to the leadership and revised as indicated. The survey instrument was then approved by project leaders.

3. Explain how the survey will be conducted. How will the customers be sampled (if fewer than all customers will be surveyed)? What percentage of customers asked to take the survey will respond? What actions are planned to increase the response rate? (Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to services provided by Web.)

This will be a Web-based survey. This methodology was appropriate since the graphics being tested will be used on the NWS website to communicate hurricane forecast information on storm surge and wind.

ERG will obtain a list of email addresses for use in a survey using a reputable survey email list vendor. The list will include only email addresses whose owners have agreed to receive further emails on topics such as this. ERG will select a sample from this list and send a series of emails to the potential respondents: pre-notification, email survey link, follow-up reminder. The set of emails to be sent are further detailed in Section B below. The survey will be a link within the email sent. ERG will host the survey on its secure Web-based survey Web site running Vovici™ survey software. All responses will be anonymous.

4. Describe how the results of this survey will be analyzed and used. If the customer population is sampled, what statistical techniques will be used to generalize the results to the entire customer population? Is this survey intended to measure a GPRA performance measure? (If so, please include an excerpt from the appropriate document.)

The statistical approach being used is described in Section B below. NWS has selected a simple random sample design and data will be extrapolated to the population of Atlantic and Gulf coastal areas from Maine to Texas, using appropriate sampling weights (see Section B).

The data will be analyzed to identify trends among the respondents related to hurricane experience and concerns and to assess whether statistically significant differences exist between demographic and geographic groupings, with particular attention paid to potentially high-risk groups, both in terms of location and social vulnerability. The data gained from this survey will be reported to the NWS project leadership with the goal of improving the communication of hurricane forecasts, particularly related to storm surge, and thus contributing to the NOAA goal of preserving life.

Part B: Collections of Information Employing Statistical Methods

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.

The purpose of the survey, as discussed above, will be to collect feedback on hurricane and wind-related information and prototype graphics being considered for possible use in communicating hurricane forecasts on NWS websites. The population for the survey will be residents of Atlantic and Gulf coastal areas from Maine to Texas.¹ The sample will be divided into two sub-samples based on location relative to the coast. A sub-sample living near the coast will be asked to complete questions about both storm surge and wind. In order to capture the opinions of those living further inland regarding the wind products, the second sub-sample will be chosen further inland.

NWS will use data in the database of email addresses to define the two sub-samples. The email data being used for this will contain zip codes associated with each email address. The zip codes will be categorized as either “near coast” (e.g., within 10 miles of the coast) and “further inland” (e.g., between 10 and 50 miles of the coast).

To estimate the potential population size, we use the populations of coastal counties² adjusted for the percentage of the population 21 and older. We expect that the graphics will be most useful to adults (e.g., over 21). Table B-1 includes estimates of the adult (over 21) population in each state living in coastal counties.

NWS is targeting 1,500 responses to this survey and will select email addresses so that they are distributed across states based on historical storm frequency. Table B-1 includes the number of storms that have made landfall in each state in the study region from 1851-2004. The sample size for each state is calculated by multiplying the 1,500 sample size by the percentage of storms for each state.

Response Rate. NWS is implementing this as a Web-based (email) survey since the graphics being assessed will all be Web-based graphics. NWS expects that of the valid email addresses, the response rate will be close to 60 percent. This response rate is based on previous Web surveys conducted by NWS and also taking into account that the survey will allow potential respondents to preview graphics that will be widely used.

Table B-1. Coastal County Population Estimates and Survey Sample Sizes by State

State	Coastal County Population [a]	Percentage of Population Over 21 [b]	Estimated Population of Coastal Counties Over 21 [c]	Historical Storm Frequency [d]		Estimated Sample to be Derived from State [e]
				Number	Pct.
Alabama	764,613	71.4%	545,934	22	5.4%	80
Connecticut	3,574,097	73.0%	2,609,091	10	2.4%	36
Delaware	897,934	72.4%	650,104	2	0.5%	7
Florida	18,427,589	74.2%	13,673,271	110	26.8%	401
Georgia	1,043,009	69.4%	723,848	20	4.9%	73
Louisiana	3,573,854	70.4%	2,515,993	49	11.9%	179
Maryland	5,287,553	72.3%	3,822,901	2	0.5%	8
Massachusetts	6,318,177	73.7%	4,656,496	10	2.4%	36
Mississippi	628,502	74.3%	466,977	15	3.6%	55
New Jersey	8,683,202	72.7%	6,312,688	2	0.5%	8
New York	17,586,787	73.2%	12,873,528	12	2.9%	44
North Carolina	2,254,172	75.9%	1,710,917	46	11.2%	168
Rhode Island	1,052,567	73.3%	771,532	9	2.2%	33
South Carolina	1,932,243	76.3%	1,474,301	31	7.5%	113
Texas	8,287,623	77.6%	6,431,195	59	14.4%	215
Virginia	5,425,647	72.5%	3,933,594	12	2.9%	44
Totals	85,737,569		63,172,370	411	100%	1,500

[a] Data for coastal counties are taken from NOAA’s State of the Coast data and web site (http://stateofthecoast.noaa.gov/population/welcome.html).

[b] Data for percentage of population over 21 years of age are taken from the Census Bureau’s American Fact Finder data (http://factfinder2.census.gov/faces/nav/jsf/pages/index.xhtml) based on data from the American Community Survey.

[c] Calculated by multiplying the coastal county population by the percentage of population over 21.

[d] Taken from NOAA Technical Memorandum NWS TPC-4, The Deadliest, Costliest, and Most Intense United States Tropical Cyclones From 1851 To 2004 (And Other Frequently Requested Hurricane Facts), Updated August 2005, Table 10.

[e] Calculated by distributing the 1,500 sample size using the percentage in the preceding column.

2. Describe the procedures for the collection, including: the statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Statistical Method for Stratification, Accuracy, and Sample Selection

The sample size for this project was determined on selecting a simple random sample for assessing a yes/no question. The basic formula for that sample size (n) determination is

where z is the standard normal distribution value for a 95 percent confidence interval (=1.96), p is the assumed value the response distribution (=50%),³ and D is the acceptable margin of error (accuracy). For this survey, NWS is seeking relatively accurate estimates for the data being collected and has specified the margin of error to be 2.5 percentage points. Calculating this formula using the parameter values specified in this paragraph results in a sample size of approximately 1,500 people.

The sample will also be stratified by distance to the coast (“near coast” and “further inland”; see response to question B.1 above) with a target of one half of the total (i.e., 750) coming from each sub-sample.

Sample units will be randomly selected from the sample frame using a systematic sampling approach within each state. Specifically, NWS will calculate a sampling frequency as the ratio (k) of total sampling frame units to desired sample units for each state. NWS will then sort the sample frame by “near coast” and “further inland” zip codes and select a random starting point within the first k units of the sorted frame and then select every kth unit thereafter.

To account for nonresponse, NWS will select a sample size that is marked up by the nonresponse rate. That is, we have assumed that response rate will be 60 percent. Thus, our sample will be 3,750 (=1,500/(1-0.6)) to account for nonresponse.

Estimation Procedure

Population parameters will be estimated by appropriately weighting the sample responses. NWS is using a probability proportional to size sampling design, making weighting straight forward. NWS will calculate population parameters by appropriate weighting each sample respondent by the inverse of their selection probability.

Unusual Problems Requiring Specialized Sampling Procedures

None are required.

Use of Periodic (Less Than Annual) Data Collection

This request is for a one-time data collection.

3. Describe the methods used to maximize response rates and to deal with nonresponse. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield "reliable" data that can be generalized to the universe studied.

Despite the expectation that a high response rate is achievable, NWS will follow good survey practices, including the following:

• NWS will send the potential respondents a pre-notification email to inform them of the upcoming survey. The email will describe the importance of collecting this information and will be clear that the survey is being performed by NWS.

• NWS will send the potential respondents an email with a link to the survey. The email will contain a description that describes the importance of the survey information and informs them that they will be providing input into the graphics that NWS will be using to convey storm information. This email will be sent 2-3 days after the pre-notification email.

• For those not responding after 5 days of receiving the survey email (previous bullet), NWS will send a reminder email with a link to the survey.

NWS will also perform a nonresponse analysis to assess the extent to which nonresponse bias may influence the resulting data. This analysis will involve comparing the demographics from this survey to demographics compiled by NOAA’s National Ocean Service’s Population Trends Along the Coastal United States: 1980-2008.⁴ This will allow NWS to determine whether the demographics in our survey match the demographics of coastal populations in general. If our sample is not representative of the general demographics, NWS will limit the scope of inferences made and will not extrapolate to the population of interest. Nevertheless, even if the data are not representative of coastal populations (based on demographics), the data and information from this survey would be valuable in assisting NWS in refining the storm risk graphics.

4. Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved OMB must give prior approval.

In preparation for the survey, NWS’ consultant Dr. Betty Morrow conducted several one-on-one webinars with less than nine citizens from Lee County, Florida, to assess their interpretation of the proposed graphics. Some graphics were also displayed for informal viewing at the American Meteorological Society’s Weatherfest, which is open to the public. Based on these discussions, as well as findings from a literature review related to public understanding of hurricane visuals, the graphics and questions were adjusted. These revised graphics were then presented at a National Hurricane Center workshop, where 80 emergency managers participated in polling their preferences on the graphics (OMB #0690-0030). NWS personnel have adjusted elements of the graphics based on this polling.

5. Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

NWS contracted with Eastern Research Group, Inc. (ERG) of Lexington, MA, and its consultants, to design the survey instrument, develop the sampling approach, implement the survey, and analyze the resulting data collected. The survey design team included the following individuals:

• Dr. Lou Nadeau (781)1- 674-7316; [email protected]).

• Dr. Betty Morrow (305) 385-5953) [email protected])

• Dr. Gina Eosco (781) 704-4458; [email protected])

1 Table B-1 includes a list of the in-scope states.

2 NOAA’s list of coastal counties, and the method for identifying coastal counties, can be found in http://www.census.gov/geo/landview/lv6help/coastal_cty.pdf. This survey would use the counties in Maine to Texas in the referenced document.

3 Using a 50% assumption for the response distribution is a worst-case scenario that maximizes the variance and thus the sample size estimate.

4 http://oceanservice.noaa.gov/programs/mb/supp_cstl_population.html

7