NWS - IDSS Supporting Statement Part A

SUPPLEMENTAL QUESTIONS PART A

U.S. Department of Commerce

National Oceanic & Atmospheric Administration

DOC/NOAA Customer Surveys

Impact Decision Support Services

OMB Control No. 0648-0342

1. Explain who will be conducting this survey. What program office will be conducting the survey? What services does this program provide? Who are the customers? How are these services provided to the customer?

The proposed information collection will be led by Vankita Brown, Cindy Woods, and Mike Sowko (NWS), supported by a team of atmospheric scientists representing various regions in NOAA. The program offices responsible for the proposed information collection are the Operations Division and Performance and Evaluation Branch in the Office of the Chief Operating Officer (OCOO).

The OCOO’s Operations Division is responsible for providing Headquarters-level support to the entire field structure through the provision of objective and impact-based performance metrics, monitoring forecast and warning consistency, facilitating communications between senior leadership, core Federal partners and field offices, and focusing on Continuity of Operations (COOP). Specifically, the Division:

• provides oversight and insight to NWS service performance and ensures NWS is responsive by delivering products and information that are useful to customers and partners;

• oversees NWS service assessments and their recommendations as many affect NWS partners in the delivery of services including the media, local officials, emergency managers, and weather and climate enterprise;

• ensures optimal resources are available to cover the HQ-based NWS Operations Center as well as liaison support to partnering Federal agencies; and

• ensures that training requirements for both operations and service assessments feed into the Office of the Chief Learning Officer.

The Performance and Evaluation Branch provides the foundation for all NWS service improvement activities by measuring, analyzing, and reporting operational performance and assessing customer satisfaction with service delivery. The Branch provides leadership and establishes policy for the verification, service assessment, customer satisfaction and forensic services programs. Specifically, the Performance and Evaluation Branch:

• develops the requirements for a robust verification system and oversees the system’s development and maintenance;

• uses teams established at each NWS service delivery point to assess needed improvements;

• uses assessment findings to recommend national changes to improve service programs;

• organizes and deploys teams to assess the level of service performance and make recommendations for increased effectiveness;

• creates customer satisfaction indices and assists service program managers in translating results into service improvements;

• establishes policy for archiving NWS data and products that are used by the public, weather sensitive economic sectors and academia for both research and legal purposes; and

• maintains the NWS natural hazards database - the only official repository for natural hazard statistics - used by agencies such as FEMA, the EPA, USGS, and organizations such as the Institute for Business and Home Safety to suggest mitigation strategies.

NWS provides Impact Decision Support Services (IDSS) to core partners when weather, water, or climate has a direct impact on the protection of lives and livelihoods. IDSS is defined as the provision of relevant information and interpretive services to enable core partners’ decisions.

IDSS may be characterized as either episode or recurring. Episodic or event-driven IDSS is support provided to core partners for a particular event/incident (ex., Webinars, NWSchat, on-site deployment). Recurring or routine IDSS is ongoing support provided to core partners throughout the year to improve partner mitigation, preparation, response, and recovery related to event/incidents, or to support routine-high value decisions (ex., joint training, scenario planning, table-top exercises, and daily coordination regarding routine high-value decisions).

The customer or core partner (who receives IDSS) is defined as a government and non-government entity who is directly involved in the preparation, dissemination, and discussions involving weather, water, or climate related NWS information that supports decision making for routine or episodic, high impact events. Such partners include emergency managers, media, public officials, and federal/state/local government.

Feedback from core partners on these specialized services is needed in order to evaluate NWS’ performance for recurring and significant events to improve services to the nation.

2. Explain how this survey was developed. With whom did you consult during the development of this survey on content? Statistics? What suggestions did you get about improving the survey?

The proposed research was planned in close consultation with a team of NWS meteorologists and social scientists with extensive expertise in the full array of weather event types. A high level of effort was expended to propose a list of questions that were informed by the cleared questions from the NOAA Generic Clearance, OMB Control No. 0648-0342. The NWS survey team identified areas in which the survey questions possessed certain limitations in question clarity, order, variable measurement, and ability for participants to meaningfully respond to question items. Thoughtful effort has been made to keep the proposed surveys as brief as possible and to ensure relevance to the primary services offered.

An earlier, OMB-approved version of the survey instrument was administered to core partners for three months in 2019. Analysis from this first test yielded positive data about the episodic or event-driving IDSS and trust in NWS staff and services, but the instrument was not sufficient in capturing data about the recurring or routine IDSS. In light of these findings, adjustments, including deleting some questions and adding additional items that more adequately measure concepts, were included.

Likewise, a pre-test was also done with a small sample of the target audience for elements including format, comprehension, readability, ease of completion, and estimated response time. Additionally, pilot participants were also interviewed to further assess the instrument to refine clarity, and identify additional concepts to test. The pre-test resulted adding page numbers, allowing the ability to scroll back to a previous question, and adding in a progress bar.

3. Explain how the survey will be conducted. How will the customers be sampled (if fewer than all customers will be surveyed)? What percentage of customers asked to take the survey will respond? What actions are planned to increase the response rate? (Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to services provided by Web.)

Participants of this data collection represent a very discreet population - emergency managers, media, public officials, and federal/state/local government partners who work closely with Weather Forecast Offices (WFO) throughout the six NWS regions. Since this work is based on evaluating IDSS to these targeted partners, forecasters will disseminate the electronic questionnaire (informed consent will be requested before respondents can advance to survey questions; see survey introduction for informed consent statement) to participants within their respective County Warning Area (CWA) only after hazardous weather events (i.e., tornadoes, hurricanes, floods, etc.). This same approach was used during the pilot testing of the instrument and based on initial responses, we anticipate a 30% response rate from participants.

4. Describe how the results of this survey will be analyzed and used. If the customer population is sampled, what statistical techniques will be used to generalize the results to the entire customer population? Is this survey intended to measure a GPRA performance measure? (If so, please include an excerpt from the appropriate document.)

Analysis of survey data will be undertaken through basic descriptive statistics only (e.g., percent, mean scores). This information collection seeks to assess the quality of IDSS and to better serve the information needs of our core partners. The respondent universe will include a sampling of our core partners. A core partner is a “government or nongovernment entity directly involved in the preparation, dissemination, and discussions involving weather, water, or climate related National Weather Service information, that supports decision making for routine or episodic, high impact events.” Core Partners include emergency managers, media, public officials, and federal/state/local government. This audience is clearly defined, and the sample will be include members of the community/region. The research team will rely heavily on NWS WFOs within their regions to aid in identifying core partners who have received IDSS.

2