NWS IDSS Supporting Statement A

SUPPLEMENTAL QUESTIONS PART A

U.S. Department of Commerce

National Oceanic & Atmospheric Administration

NWS Customer Surveys

Impact Decision Support Services

OMB Control No. 0648-0342

1. Explain who will be conducting this survey. What program office will be conducting the survey? What services does this program provide? Who are the customers? How are these services provided to the customer?

The National Weather Service (NWS) will be conducting this survey on a situational basis after significant weather events. The survey will be sent from NWS National, Regional, and Local offices to the Core Partners of the respective offices. A Core Partner is defined as a “government or nongovernment entity directly involved in the preparation, dissemination, and discussions involving weather, water, or climate related National Weather Service information, that supports decision making for routine or episodic, high impact events.”

The proposed information collection will be led by Katherine Edwards, Andy Foster, James Morrow, and Joseph Moore (NWS), supported by a team of atmospheric scientists representing various regions and offices in NOAA’s National Weather Service. The program office responsible for the proposed information collection is the Decision Support Integration branch of the Analyze, Forecast and Support Office.

The Decision Support Integration Branch will work with the Operations Division and Performance and Evaluation Branch in the Office of the Chief Operating Officer (OCOO), that is responsible for providing Headquarters-level support to the entire field structure through the provision of objective and impact-based performance metrics, monitoring forecast and warning consistency, facilitating communications between senior leadership, core Federal partners and field offices, and focusing on Continuity of Operations (COOP). Specifically, the Division:

• provides oversight and insight to NWS service performance and ensures NWS is responsive by delivering products and information that are useful to customers and partners;

• oversees NWS service assessments and their recommendations as many affect NWS partners in the delivery of services including the media, local officials, emergency managers, water resource managers and weather and climate enterprise;

• ensures optimal resources are available to cover the HQ-based NWS Operations Center as well as liaison support to partnering Federal agencies; and

• ensures that training requirements for both operations and service assessments feed into the Office of the Chief Learning Officer.

The Performance and Evaluation Branch provides the foundation for all NWS service improvement activities by measuring, analyzing, and reporting operational performance and assessing customer satisfaction with service delivery. The Branch provides leadership and establishes policy for the verification, service assessment, customer satisfaction and forensic services programs. Specifically, the Performance and Evaluation Branch:

• develops the requirements for a robust verification system and oversees the system’s development and maintenance;

• uses teams established at each NWS service delivery point to assess needed improvements;

• uses assessment findings to recommend national changes to improve service programs;

• organizes and deploys teams to assess the level of service performance and make recommendations for increased effectiveness;

• creates customer satisfaction indices and assists service program managers in translating results into service improvements;

• establishes policy for archiving NWS data and products that are used by the public, weather sensitive economic sectors and academia for both research and legal purposes; and

• maintains the NWS natural hazards database - the only official repository for natural hazard statistics - used by agencies such as FEMA, the EPA, USGS, and organizations such as the Institute for Business and Home Safety to suggest mitigation strategies.

NWS provides Impact Decision Support Services (IDSS) to Core Partners when weather, water, or climate has a direct impact on the protection of lives and livelihoods. IDSS is defined as the provision of relevant information and interpretive services to enable Core Partners’ decisions.

IDSS is either episodic or recurring. Episodic or event-driven IDSS is support provided to Core Partners for a particular event/incident (ex., Webinars, NWSchat, on-site deployment). Recurring or routine IDSS is ongoing support provided to Core Partners throughout the year to improve partner mitigation, preparation, response, and recovery related to event/incidents, or to support routine-high value decisions (ex., joint training, scenario planning, table-top exercises, and daily coordination regarding routine high-value decisions).

The customer or Core Partner (who receives IDSS) is defined as a government and non-government entity who is directly involved in the preparation, dissemination, and discussions involving weather, water, or climate related NWS information that supports decision making for routine or episodic, high impact events. Such partners include emergency managers, media, public officials, and federal/state/local/territorial/tribal/ government.

Feedback from Core Partners on these specialized services is needed in order to evaluate NWS’ performance for recurring and significant events to improve services to the nation.

2. Explain how this survey was developed. With whom did you consult during the development of this survey on content? statistics? What suggestions did you get about improving the survey?

This survey was refined from previous external surveys (OMB Control No. 0648-0342) performed by the NWS Program Management Office Impact-based Decision Support Services (IDSS) Measurements team as well as NWS Office of Organizational Excellence (OOE) funded project to develop a Customer Experience Roadmap and initial metrics recommendations.

The proposed collection was planned in close consultation with a team of NWS meteorologists and social scientists with extensive expertise in the full array of weather event types. The NWS survey team identified areas in which the survey questions possessed certain limitations in question clarity, order, variable measurement, and ability for participants to meaningfully respond to question items. Thoughtful effort has been made to keep the proposed surveys as brief as possible and to ensure relevance to the primary services offered.

An earlier, OMB-approved version of the survey instrument (OMB Control No. 0648-0342) was administered to Core Partners for three months in 2019. Analysis from this first test yielded positive data about the episodic or event-driving IDSS and trust in NWS staff and services, but the instrument was insufficient in capturing data about the provision of routine IDSS. In light of these findings, adjustments, including deleting some questions and adding additional items that more adequately measure concepts, were included.

Likewise, a pre-test was also done with a small sample of 8 people from the target audience for elements including format, comprehension, readability, ease of completion, and estimated response time. Additionally, pilot participants were also interviewed to further assess the instrument to refine clarity, and identify additional concepts to test.

3. Explain how the survey will be conducted. How will the customers be sampled (if fewer than all customers will be surveyed)? What percentage of customers asked to take the survey will respond? What actions are planned to increase the response rate? (Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to services provided by Web.)

Participants of this data collection represent a very discreet population - emergency managers, water resource managers, media, public officials, and federal/state/local government partners who work closely with Weather Forecast Offices (WFO) throughout the six NWS regions and National Centers for Environmental Prediction. Since this work is based on evaluating IDSS to these targeted Core Partners, operational units will disseminate the electronic questionnaire (informed consent will be requested before respondents can advance to survey questions; see survey introduction for informed consent statement) to participants within their respective County Warning Area (CWA) or area of coverage only after hazardous weather events (i.e., tornadoes, hurricanes, floods, etc.). This same approach was used during the pilot testing of the instrument and based on initial responses, we anticipate a 30% response rate from participants.

4. Describe how the results of this survey will be analyzed and used. If the customer population is sampled, what statistical techniques will be used to generalize the results to the entire customer population? Is this survey intended to measure a GPRA performance measure? (If so, please include an excerpt from the appropriate document.)

Analysis of survey data will be undertaken through basic descriptive statistics (e.g., percent, mean scores). This information collection seeks to assess the quality of IDSS and to better serve the information needs of our Core Partners. The respondent universe will include a sampling of our Core Partners. A Core Partner is a “government or nongovernment entity directly involved in the preparation, dissemination, and discussions involving weather, water, or climate related National Weather Service information, that supports decision making for routine or episodic, high impact events.” Core Partners primarily include emergency managers, water resource managers, media, public officials, and federal/state/local government. The Core Partner audience is clearly defined in NWS policy (https://www.nws.noaa.gov/directives/sym/pd01024curr.pdf). The research team will rely heavily on NWS WFOs within their regions to aid in identifying Core Partners who have received IDSS.