Supplemental and Part B Questions

1. Supplemental Questions for DOC/NOAA Customer Survey Clearance
   (OMB Control Number 0648-0342)

1. Explain who will be conducting this survey. What program office will be
   conducting the survey? What services does this program provide? Who are the customers? How are these services provided to the customer?

NOAA’s National Weather Service (NWS) and National Hurricane Center (NHC) seek to gather feedback via a Web-based survey from the public on the NHC’s experimental Arrival of Tropical Storm Force Winds graphic. This study is focused on the general public and their interpretation, understanding, and use of the experimental graphic. The surveyed public includes coastal states along the Gulf and Atlantic coasts, as well as Washington, DC, Hawaii, and the Virgin Islands [Puerto Rico removed from sample strategy due to storm damage from Hurricane Maria and availability to respond to an online survey].

The anticipated arrival of sustained tropical storm force winds from a tropical cyclone is important for coastal communities and the public to know when to prepare their homes or businesses. Once sustained tropical storm force winds begin, such preparations usually become too dangerous or difficult. Previously, many decision makers have inferred the arrival of sustained tropical storm force winds from NHC products deterministically, without accounting for the tropical cyclone track or size uncertainty. The risk in not factoring in these uncertainties is that communities may have less time to prepare if a tropical cyclone speeds up or increases in size beyond NHC’s initial forecasts. To better meet users’ needs, NHC has developed a set of prototype graphics that depict when sustained tropical storm force winds from an approaching tropical cyclone could arrive at individual locations. There are two versions of the experimental arrival of tropical storm force winds graphic available on the NHC website for all tropical cyclones, post-tropical cyclones, and potential tropical cyclones for which NHC is issuing advisories:

1. Earliest Reasonable Arrival Time: the primary graphic, which identifies the time window that users at individual locations can safely assume will be free from tropical storm force winds.

2. Most Likely Arrival Time: the graphic that identifies the time before or after which the onset of tropical storm force winds is equally likely.

Timing information will be available for locations that have at least a 5% chance of experiencing sustained tropical storm force winds during the next 5 days. The graphics will be updated with each new NHC full advisory package. Arrival times will be depicted with higher temporal resolutions (i.e., in 6-hour intervals) during the first day of the 5-day forecast, increasing to lower temporal resolution (i.e., in 12-hour intervals) after the first day of the 5-day forecast period.

2. Explain how this survey was developed. With whom did you consult during the development of this survey on content? statistics? What suggestions did you get about improving the survey?

The NWS contracted with Abt Associates (Abt) on the development and deployment of the survey. To develop the survey, Abt worked with Responsive Management, a public opinion and attitude survey research firm specializing in environmental issues, including the development and testing of surveys, implementing surveys, and statistical analysis and reporting. Abt and Responsive Management worked closely with Robbie Berg, a forecaster with the NHC, to develop the survey and the prototype graphics. Suggestions for improving the survey focused on the following:

• Ensuring the questions are more broadly focused on preparations for sustained winds and not specifically evacuations.

• Rephrasing questions from “predicted arrival time” to accurately portray the probabilistic nature of the information. More appropriate phrasing includes the use of possible arrival and anticipated arrival.

• The prototype graphics developed by NHC include five regional storms with an anchor point. Questions were rephrased to address the interpretation for a given regional graphic rather than one’s permanent residential location.

• Ensured questions did not incorrectly state “predicted wind speed,” which is not information provided in this product.

3. Explain how the survey will be conducted. How will the customers be sampled (if fewer than all customers will be surveyed)? What percentage of customers asked to take the survey will respond? What actions are planned to increase the response rate?

This survey will be conducted online. The experimental version of the tropical storm force winds graphical web versions will be integrated into the online survey. The States and Territories to be sampled are listed in Table 1. The sampling plan consists spreading a sample size of approximately 1496 across 22 States and Territories.

The sample panel of respondents will be developed according to the latest U.S. Census population data. Residents will be sampled using zip code and/or county delineations so that each state/territory sample is representative of both the coastal and inland regions of the state/territory. The survey results will be reflective of the population of residents in each area ages 18 years old and older.

Potential survey respondents will be asked to participate via an email invitation and will be provided a direct link to the online survey. The email addresses for the sample will be obtained from the two premier sample providers, Survey Sampling International (SSI) and Marketing Systems Group (MSG). Respondents can then complete the survey at any time of the day or day of the week. Unique access codes are used to ensure that each respondent completes the survey only once.

This online survey will be conducted with people who have agreed to be surveyed, which supports an increased response rate. Online surveys generally have response rates estimated to be about 25%; however, calculating an exact response rate for online sampling is nearly impossible, as there is no way to determine with certainty if an email address is valid. Unlike a telephone survey, where an interviewer can indicate if a number tried has been disconnected, in most instances non-valid email addresses do not have a provision to alert the sender that the email is no longer valid.

The questionnaire was designed with response rates in mind to increase the response rate. The questionnaire was developed to entail no more than approximately 10 minutes of respondent time. Internal testing and live pretesting of surveys that Abt and Responsive Management have conducted have suggested that surveys longer than about 10 minutes cause considerable respondent drop-out. Therefore, this survey was designed to be no more than 10 minutes. Contact with potential respondents is also important. After the initial email invitation to the survey is sent, reminder emails are sent to those who have not responded to the survey. These reminders encourage non-participants to take the survey. Further, the survey will be available in both English and Spanish, thereby increasing the likelihood that people who are either non-English speaking or who prefer Spanish will take the survey. Finally, the survey has been designed as mobile friendly to increase response rate.

Table 1. List of all States and Territories to be surveyed and the summarized data assessed for the sampling strategy.

Region	State	Coast Length (mi)	Population 2010
Gulf	Texas	4,562	6,418,288,662
	Louisiana	7,806	288,469,440
	Mississippi	308	245,728,170
	Alabama	437	317,284,731
Southeast	Florida	11,295	1,267,480,004
	Georgia	742	1,592,233,155
	South Carolina	1,616	213,888,454
	North Carolina	4,870	955,205,400
Northeast	Maryland	4,289	137,541,408
	Virginia	4,252	1,067,401,254
	District of Columbia	38	600,671
	Delaware	412	2,681,172
	Pennsylvania	35	842,485,269
	New Jersey	1,706	185,269,833
	New York	2,011	1,211,711,322
	Connecticut	581	28,286,296
	Rhode Island	437	5,292,060
	Massachusetts	1,621	91,780,304
	New Hampshire	155	13,299,150
	Maine	4,123	21,418,320
Other	Virgin Islands	320	106,405
	Hawaii	1,277	6,547,900

4. Describe how the results of this survey will be analyzed and used. If the customer population is sampled, what statistical techniques will be used to generalize the results to the entire customer population? Is this survey intended to measure a GPRA performance measure? (If so, please include an excerpt from the appropriate document.)

NOAA/NWS/NHC will use the information resulting from this testing to help guide refinements to the Arrival of Tropical Storm Force Winds Graphic. Thus, data from this survey will contribute to the goal of improving the communication of hurricane forecasts, thus contributing to the NOAA goal of preserving life and property. The NWS is not using any statistical methods to select participants from the population, and the data do not directly contribute to a GPRA measure.

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.

The potential respondent universe is the population of all the states/territories in the study area (Table 1). The “unit” to be studied in this survey is the person (18 years old and older), not the household. The sampling across States and Territories is equally distributed. This sampling consists of 68 total samples per state/territory with an equal expected response rate of 25% (Table 2). The final response rate (estimated) can be generated only after the survey has been completed. Calculating an exact response rate for online sampling is nearly impossible, as there is no way to determine with certainty if an email address is valid. In making the calculation, this can be handled in two ways to arrive at either an “estimated” response rate or a “worst-case-scenario” response rate:

1) A certain percentage of unanswered emails will be assumed to be non-valid, and that percentage will be deducted from the sample before the response rate is calculated. This provides an estimated response rate; OR

2) No deduction will be made for assumed non-valid emails, and the response rate will be calculated on the total number of addresses contacted, with knowledge that the actual response rate is better than the calculated response rate. In other words, this calculation provides a lowest possible response rate (a worst-case-scenario), or a bottom, that the actual response rate exceeds.

See Table 2 for the breakdown of respondents by States and Territories. The expected response rate for the collection is 25% (374 completed surveys out of a total sample universe of 1496 participants)

Table 2. The estimated response rate by State and Territory.

Region	State	Total Sample	Expected Response	Response Rate
Gulf	Texas	68	17	25%
	Louisiana	68	17	25%
	Mississippi	68	17	25%
	Alabama	68	17	25%
Southeast	Florida	68	17	25%
	Georgia	68	17	25%
	South Carolina	68	17	25%
	North Carolina	68	17	25%
Northeast	Maryland	68	17	25%
	Virginia	68	17	25%
	District of Columbia	68	17	25%
	Delaware	68	17	25%
	Pennsylvania	68	17	25%
	New Jersey	68	17	25%
	New York	68	17	25%
	Connecticut	68	17	25%
	Rhode Island	68	17	25%
	Massachusetts	68	17	25%
	New Hampshire	68	17	25%
	Maine	68	17	25%
Other	Virgin Islands	68	17	25%
	Hawaii	68	17	25%

2. Describe the procedures for the collection, including: the statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Statistical Method for Stratification and Sample Selection

The plan called for a specified number of people in each state (68 completed questionnaires) so that there would be enough sample in each state for reasonable statistical validity. Because each state has a set goal of number of completed questionnaires to be obtained, regardless of the proportion of the total population attributed to that particular state, the overall data are weighted so that each state ends up being in the correct proportion for overall data.

Estimation Procedure and Accuracy

The sample is designed to be representative of the population of the participating states as whole by weighting the data based on the state population so that the overall data are in the proper proportions by state. Percentages in the survey can then simply be multiplied by the total population to arrive at estimates of the number of people in the general population who hold a particular view. The degree of accuracy will be higher on the overall data than in each state. In the overall sample, there will be 1,496 completed questionnaires, which will provide a sampling error of less than 5%. The sampling error in each of the 22 states will vary based on the state’s coastal population but will be less than 10% in any given state.

Unusual Problems Requiring Specialized Sampling Procedures

None are required.

Periodic Data Collection Cycles

This request is for a one-time data collection.

3. Describe the methods used to maximize response rates and to deal with nonresponse. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield "reliable" data that can be generalized to the universe studied.

Response rates and nonresponse bias are different (although they can be related). Nonresponse bias refers to the situation where non-responders are different from responders. When this is true, then the response rate becomes more important. When this is not true, then the response rate is of less importance. The way to address nonresponse bias is to first obtain a good response rate. Ways to do this for this survey include: (1) survey designed to entail no more than approximately 10 minutes of respondent time; (2) reminder emails sent after initial invitation; (3) survey langauges include English and Spanish; and (4) mobile friendly format to complete the survey via a mobile devise.

Secondly, once stringent efforts have been made to achieve a good response rate, the final data are then checked to ensure that the final sample of people who took the survey is representative of the general population under study. This representation is fine-tuned by weighting the final data to match the age and gender breakdown of the actual population of the area under study (in this case, the 22 states under study). This ensures that demographic groups with higher non response are still fully represented. Because a good response rate will have been achieved (based on the efforts previously described to encourage participation in the survey), the necessary weighting will be minimal. This weighting thus matches the final data to the actual population but does so in such a way that no single demographic group is weighted unacceptably high but all demographic groups, including those less likely to respond, are represented in the final data.

4. Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved OMB must give prior approval.

An internal review has been conducted, and the survey pretested to reveal potential obstacles or unclear components. The survey has been pretested by two staff not working on the project with a focus on how long it took to complete the survey as well as feedback for improving the survey, such as refining any questions or instructions that were unclear or hard to follow. The feedback received has also been reviewed to ensure that the survey is addressing the goals and objectives of the study.

5. Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The NWS has contracted with Abt Associates of Cambridge, MA to design the survey, implement data collection, and to develop a report. Abt’s project manager for this work is Dr. Susan Taylor (301-634-1755; [email protected])