0648-0797 Supporting Statement Part A

SUPPORTING STATEMENT

U.S. Department of Commerce

National Oceanic & Atmospheric Administration

Tornado Watch/Warning Post-Event Evaluation

OMB Control No. 0648-0797

SUPPORTING STATEMENT PART A

Abstract

This is a request for a revision and extension of the OMB control number 0648-0797: Tornado Watch/Warning Post-Event Evaluation.

The NOAA National Severe Storms Laboratory (NSSL) designed the data collection instrument to allow for more routine collection of this information. This survey is delivered through a web application hosted by NSSL called Tornado Tales, available online at https://inside.nssl.noaa.gov/tornado-tales/. After approval of our initial data collection instrument (that shown on the website), researchers carried out post-event data collections for multiple tornado events, validating the questions and identifying issues for improvement. This fieldwork led to several needed improvements, including the addition of questions about the event more broadly, rephrasing some questions, and including questions about efficacy and the availability of forecast information. While these revisions have added questions to the survey, their improved clarity should allow for faster response times per question. We estimate the time to complete the survey is five to ten minutes on average. Subject recruitment will primarily be done by NOAA NSSL and its partners advertising the survey via websites and social media outlets. In addition to the changes to the survey instrument, researchers at NOAA NSSL and the University of Oklahoma would like to conduct interviews with emergency managers, broadcast meteorologists, and members of the public after certain tornado events to explore how they interpreted and used NOAA forecast information. Researchers will use a skip-logic approach, meaning participants will only answer questions about the time periods relevant to their personal experience. Given the in-person nature of these interviews, we expect them to take between 15 and 30 minutes on average.

Justification

1. Explain the circumstances that make the collection of information necessary. Identify any legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the collection of information.
   
   Congress established Social Behavioral and Economic Sciences (SBES) policies for weather forecasting in the Weather Research and Forecasting Innovation Act (WRFIA) of 2017. The legislation specified, at a high level, that the purpose of the NOAA watch/warning system is to inform action to prevent loss of life and property (Sec. 406), and that any changes to the system must preserve the benefits of the current system and be guided by the findings of social scientific research.  NOAA responded to this guidance in its 2020-2026 R&D Vision, calling out reductions in societal impacts from hazardous weather in Vision Area #1.  Key Question 1.4 asks, “How can NOAA enhance communications, products, and services to enable informed decision-making?”  The vision document prescribes that this should be done by first understanding the risk communication process that supports decision-making by the public.  Optimizing NOAA’s communication outcomes requires that messages are widely disseminated, understood, and actionable.
   
   Another important policy set forth in WRFIA (Sec. 103) is the Tornado Warning Improvement and Extension Program (TWIEP).  The goal for this program is “to reduce the loss of life and economic losses from tornadoes through the development and extension of accurate, effective, and timely tornado forecasts, predictions, and warnings, including the prediction of tornadoes beyond 1 hour in advance.”  The TWIEP Implementation Plan submitted to Congress by NOAA (2017) includes several key areas of SBES research and development, including research into the ways gap-filling probabilistic information could be generated, disseminated, used by core partners, and ultimately applied to inform decision-making on the part of the U.S. public.  To evaluate the success of this new technology and assure that any changes maintain the benefits of the current watch/warning system (WRFIA Sec. 406), we need baseline data regarding the performance of the current system.

2. Indicate how, by whom, and for what purpose the information is to be used. Except for a new collection, indicate the actual use the agency has made of the information received from the current collection.

The data to be collected includes a small set of questions relating to tornado forecast reception, understanding, and response. The survey is estimated to take five to ten minutes to complete. The participants will be any member of the U.S. public that has been in or near a recent tornado event. Surveys will be administered via a webpage at https://inside.nssl.noaa.gov/tornado-tales/. Data is stored in a password-protected database and all responses are anonymous. The dataset has been used to evaluate where and how people receive, understand, and respond to information about a tornado event, resulting in more useful and efficient communication of tornado forecasts.

Interviews will be conducted in-person and recordings will be transcribed by local software. Once transcriptions are complete and de-identified, recordings will be deleted. Transcriptions will be stored on a password-protected machine. These interviews will be used to more fully evaluate why people make the decisions they do leading up to and during tornado events beyond what a survey instrument can collect.

3. Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology, e.g., permitting electronic submission of responses, and the basis for the decision for adopting this means of collection. Also, describe any consideration of using information technology to reduce burden.

The survey instrument will be delivered via an NSSL web tool. This web tool will rely on voluntary submissions when tornadoes are experienced. Being web-based means that costs for getting to and from the field are reduced. Currently much of this information is only recorded in large impact events, meaning we are not able to understand the broader pattern of response (or not) during “smaller” or more “regular” tornado events. The electronic nature of the data collection ensures a more equitable collection of information across tornado severity and location.

4. Describe efforts to identify duplication. Show specifically why any similar information already available cannot be used or modified for use for the purposes described in Question 2

Besides this collection, the only systematic data collected about severe weather warning effectiveness comes from annual customer satisfaction surveys conducted by the National Weather Service. Those surveys do not take a post-event perspective, only capturing general thoughts on how the system performs as a whole. The questions also do not measure comprehension and real-world response. Thus, we do not have the data necessary to evaluate performance in real events as needed to satisfy the requirements of the WRFIA.

5. If the collection of information impacts small businesses or other small entities, describe any methods used to minimize burden.

This collection is directed at individuals or households only. There is no impact on small businesses or other small entities.

6. Describe the consequence to Federal program or policy activities if the collection is not conducted or is conducted less frequently, as well as any technical or legal obstacles to reducing burden.

If data collection is not conducted, NOAA will be hampered in its efforts to evaluate the societal performance of its tornado watch/warning program.

7. Explain any special circumstances that would cause an information collection to be conducted in a manner inconsistent with OMB guidelines.

This collection will be conducted in a manner consistent with OMB guidelines.

8. If applicable, provide a copy and identify the date and page number of publications in the Federal Register of the agency's notice, required by 5 CFR 1320.8 (d), soliciting comments on the information collection prior to submission to OMB. Summarize public comments received in response to that notice and describe actions taken by the agency in response to these comments. Specifically address comments received on cost and hour burden.

The notice in the Federal Register was published on February 9, 2024, in Vol. 89, No. 28 on pages 9129-9130. No comments were received during the 60-day period.

NSSL scientists consulted with external stakeholders to obtain their views on the availability of data, frequency of collection, the clarity of instructions and recordkeeping, disclosure, or reporting format (if any), and on the data elements to be recorded, disclosed, or reported. No comments were received.

9. Explain any decision to provide any payment or gift to respondents, other than remuneration of contractors or grantees.

No payment or gifts will be provided for participation.

10. Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statute, regulation, or agency policy. If the collection requires a system of records notice (SORN) or privacy impact assessment (PIA), those should be cited and described here.

While no PII will be collected, a notice of confidentiality is included, alongside a Frequently Asked Questions (FAQs) button. These are clickable and clearly displayed above the survey in a box titled ‘Privacy.’

11. Provide additional justification for any questions of a sensitive nature, such as sexual behavior or attitudes, religious beliefs, and other matters that are commonly considered private. This justification should include the reasons why the agency considers the questions necessary, the specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.

We will not collect sensitive data.

12. Provide estimates of the hour burden of the collection of information.

The number of respondents and frequency of response are hard to quantify. This is because the data collection is based on voluntary contribution of participants after a tornado has occurred. As the frequency and magnitude of tornadoes vary, the number of people impacted also fluctuates. The burden hours/response was calculated based on testing of the survey draft with colleagues. The average wage rate (in this case taken to be the total private non-farm payroll) amount of $30.17 (data taken from the US Bureau of Labor Statistics - https://www.bls.gov/news.release/empsit.t19.htm ) was used to calculate the hourly wage rate for respondents.

Information Collection	Type of Respondent (e.g., Occupational Title)	# of Respondents/year (a)	Annual # of Responses / Respondent (b)	Total # of Annual Responses (c) = (a) x (b)	Burden Hrs / Response (d)	Total Annual Burden Hrs (e) = (c) x (d)	Hourly Wage Rate (for Type of Respondent) (f)	Total Annual Wage Burden Costs (g) = (e) x (f)
1. NSSL web tool respondent costs	Any individual over the age of 18 that has experienced a tornado event	1200	1	1200	0.17 hrs (10 mins)	200	For private nonfarm payrolls= $ 30.17.	$6034.00
2. Post-event interviews	Any Emergency Manager, Broadcast Meteorologist, or member of the public who recently experienced a tornado event	50	1	50	0.5	25	For private nonfarm payrolls= $ 30.17.	$754.25
Totals				1250		225		$6788.25

13. Provide an estimate for the total annual cost burden to respondents or record keepers resulting from the collection of information. (Do not include the cost of any hour burden already reflected on the burden worksheet).

There are no capital costs or operating and maintenance costs associated with this information collection.

14. Provide estimates of annualized cost to the Federal government. Also, provide a description of the method used to estimate cost, which should include quantification of hours, operational expenses (such as equipment, overhead, printing, and support staff), and any other expense that would not have been incurred without this collection of information.

Cost Descriptions	Grade/Step	Loaded Salary /Cost	% of Effort	Fringe (if Applicable)	Total Cost to Government
Federal Oversight and data cleaning	13/1	$100,000	5	NA	$5,000
Contractor costs:
Web Development costs	Web Developer	$70,000	10	$ 1,820	$8,820
Analysis of data sets and report writing	Research Scientists/Associates	$70,000	10	$ 1,820	$8,820
TOTAL					$22,640

15. Explain the reasons for any program changes or adjustments reported in ROCIS.

The increased burden response time is due to additional questions that were added to the survey after initial testing of the instrument after three tornado events. These new questions include more data collection about respondents’ decision-making processes. These changes will help researchers understand the kinds of information people need to efficiently make decisions, which will result in better tornado forecast and warning information. Additionally, stakeholder interviews are being added to obtain better insight into why people are making the decisions they are in the days and hours before tornado events, again resulting in more effective tornado forecast and warning information.

16. For collections of information whose results will be published, outline plans for tabulation and publication. Address any complex analytical techniques that will be used. Provide the time schedule for the entire project, including beginning and ending dates of the collection of information, completion of report, publication dates, and other actions.

Findings from the survey and interviews will be collated and reported annually. The report will include basic summary statistics of each response item, with some simple analysis for different socio-demographic characteristics (when data available) and geographic area, including NWS County Warning Areas and zip codes. Data collection will continue indefinitely. Annual reports will be made available to external audiences.

17. If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons that display would be inappropriate.

The agency plans to display the expiration date for OMB approval of the collection on all instruments.

18. Explain each exception to the certification statement identified in “Certification for Paperwork Reduction Act Submissions."

The agency certifies compliance with 5 CFR 1320.9 and the related provisions of 5 CFR 1320.8(b)(3).