0648-NSSL SUPPORTING STATEMENT Part B - Final

SUPPORTING STATEMENT

U.S. Department of Commerce

National Oceanic & Atmospheric Administration

Tornado Watch/Warning Post-Event Evaluation

OMB Control No. 0648-XXXX

SUPPORTING STATEMENT PART B - (Questions and guidance for Responses)

B. Collections of Information Employing Statistical Methods

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

Potential respondents will include any member of the U.S. public that was recently (within the previous 30 days) in or near a tornado.

No records exist regarding the exact number of individuals affected by tornadoes each year in the U.S. Thus, we generate an estimate calculated on the following known information.

Based on FEMA reports and covering 11 tornadoes declared as major disasters from 2017 to 2020 (covering 58 counties in 10 states), the average number of impacted households in violent tornadoes is approximately 1000. Path length for violent tornadoes averages 10x higher than path lengths for strong tornadoes, and 100x higher than path lengths for weak tornadoes. There are approximately 1200 tornadoes per year, with 1% violent tornadoes, 19% strong tornadoes, and 80% weak tornadoes.

The average number of tornadoes by category range per year generate 12 violent, 228 strong and 960 weak tornadoes.

Furthermore, the number of households in the path of tornadoes in the US per year can be calculated using these figures. Therefore, 1000 households exposed to violent tornadoes multiplied by the average of 12 per year, gives 12,000 households. Similarly for strong tornadoes the figures are 100 households multiplied by 228, giving a total of 22,800 households, whilst 10 households exposed to weak tornadoes can be multiplied by the 960 weak tornadoes that occur, with a total of 9600 households. When adding these figures together, a total of 44,400 households exposed yearly to tornadoes

With an average of approximately 2.5 individuals per household, the total number of people exposed to tornadoes each year is estimated to be 111,000. This is likely to be an upper-bound estimate since disasters are a function of both hazard and population exposure, and thus, more dense populations are more likely to incur major damage from tornadoes.

We are using convenience sampling approaches to collect data. Those methods include:

(1) The National Weather Service Damage Assessment Tool. NWS forecasters, when evaluating structural damage after tornado events, may interview those present in/near the structures and record answers in the tool. NWS forecasters (personal communication) estimate that they encounter at least one individual in every three tornado damage surveys, thus, we estimate NWS forecasters will encounter approximately 400 individuals (⅓*1200 tornadoes).

Upon being approached by a survey or interview team, we anticipate a near 100% response rate. Previous post-event survey research has revealed that individuals are eager to share their experiences with tornadoes after events occur. To avoid duplication of surveys, the methods will use the online version when collecting in the field, or alternatively provide a QR code for respondents to fill in at their leisure.

(2) A citizen science web tool developed by the National Severe Storms Laboratory (NSSL). After tornado events, participants may access the tool and respond to the questions as a web survey. We aim to publicize this tool as well as have a QR code that will allow members of the public with a smart device to quickly access the landing page for the survey. By publicizing through social media as well as through the NSSL website, we aim to draw people to take part in the survey. Although the survey doesn’t currently exclude anyone, a computers/smart device’s IP address is used to log with the server (the IP address is only held temporarily for the allotted time before erasure), so that repeated reports and surveys cannot be submitted within a certain pre-determined time-frame. At the time of initial launch, when the tool is not yet well-known, we estimate that we will reach 1% of the tornado-affected population, or around 1,100 people, through this method. However, this is conservative and on the low side of our estimation. We are unlikely to have certainty regarding this figure, until we start to collect data.

Information Collection	Population or Potential Respondents Universe (a)	Number of Respondents Selected (b)	Expected Response Rate (c)	Expected Number of Respondents (d) = (b) x (c)
NWS DAT	All of the United States population that has experienced a tornado, and that was encountered by an NWS employee performing a damage assessment	400	100%	400
NSSL web tool	111,000	1,100	100%	1,100
Total				1,500

2. Describe the procedures for the collection of information including:

   • Statistical methodology for stratification and sample selection, No statistical sampling methods will be employed at this stage. We use a convenience sampling approach because this is an exploratory data collection exercise designed to offer new insights into post-event responses that will inform a later, more systematic investigation.

• Estimation procedure

We will generate descriptive statistics for each survey item, and eventually conduct more sophisticated regression analysis to link behaviors to demographic characteristics.

• Degree of accuracy needed for the purpose described in the justification,

There is no specific degree of accuracy required for this collection.

• Unusual problems requiring specialized sampling procedures

The collection of data after disaster events requires a special set of procedures to reach populations under difficult conditions, in particular, when they may be displaced. Our sample procedure relies on chance encounters with those who were in the path, as well as mobile/internet access for those who are not reachable at the disaster site.

• Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Data collection is anticipated to go on continuously throughout the year as tornado events occur.

An exploratory pilot study is appropriate at this stage, as little is known about the wide array of event types that take place, nor have measures like these been deployed before in any attempt to collect this data more systematically (Eldridge et al. 2016). Basic descriptives, even with a biased sample, will offer insight into latter stages of program development. This pilot study could also provide the basis needed to obtain funding to do more systematic research. We estimate that with 5-10 years of longitudinal data, we may be able to assess trends and identify gaps in knowledge that will lead to more research and development.

Reference cited:

Sandra M Eldridge, Gillian A Lancaster, Michael J Campbell, Lehana Thabane, Sally Hopewell, Claire L Coleman, Christine M Bond. Defining Feasibility and Pilot Studies in Preparation for Randomised Controlled Trials: Development of a Conceptual Framework. PLoS ONE. 2016; 11(3)

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

Participants approached by NWS forecasters will be told their participation is completely voluntary, and the NWS forecaster will appear in their official capacity, increasing trust in the data collection process. If problems are experienced with data collection through either means, our team will evaluate the nature of the issue and generate a plan for improvement. For example, NWS forecasters may find that individuals in and near damaged structures may not wish to engage in a survey about their responses at that time, and they may need to arrange another time to meet with the respondent to obtain their responses. All data collected will be used to understand the ways different groups, across different events, receive, understand, and respond to NWS watch/warning information.

Participants accessing the NSSL web tool will self-select into the study, and the protocol is very short, promoting successful completion and reduced burden (estimated time to complete the web survey is five to seven minutes, with a median of six minutes). The NSSL media team is involved in crafting messaging and media engagement to promote the NSSL web tool. The more the public that engages with the web tool following tornado events, the broader the sampling base and the more representative of impacted populations. The web tool does not require specific software or the downloading of an App, this was a purposeful decision to be as inclusive as possible and minimizing cost (through use of cell-phone data, for instance). Participants will only require web access through computers, tablets or smartphones. Links to the tool will be disseminated through social media, including outreach to broadcasters who can disseminate the link to much wider populations. If successful, this web tool will remain in place, with the public able to continue to use when tornadoes occur. This will also allow researchers to gain insight into what safety messaging is effective in initiating behavior change when tornado watches and warnings are given.

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

Periodically, the responses will be evaluated to understand the geographic scope of events receiving replies, the scope of different kinds of tornado events for which responses are obtained, and the population characteristics of responses. These measures will be compared to the total potential sample of tornado events and census tract-level demographic data to identify any biases in data collection. For example, it could be the case that respondents that volunteer this kind of information might skew older, female, and white, based on outcomes typical of mail surveys. This information will be used to inform a subsequent, more systematic (funded) investigation.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Consultants on statistical aspects of the design:

Dr. Kimberly Klockow-McClain, 405-325-0805, CIMMS/NSSL

Dr. Vankita Brown, 301-427-9338, NWS

Dr. Justin Sharpe, 405-325-5218, CIMMS/NSSL

Agency unit collecting/analyzing information:

NOAA National Weather Service

NOAA National Severe Storms Laboratory