Impact-Based Decision Support System Core Partner Survey supp questions

1. Supplemental Questions for DOC/NOAA Customer Survey Clearance
   (OMB Control Number 0648-0342)

1. Explain who will be conducting this survey. What program office will be
   conducting the survey? What services does this program provide? Who are the customers? How are these services provided to the customer?

The survey will be conducted by McKinsey and Company (a contractor) through the Office of Organizational Excellence (OOE) at the National Weather Service (NWS). The OOE is leading a project to evaluate NWS’s ability to deliver Impact-based Decision Support Services (IDSS) to external stakeholders in support of its vision of creating a Weather-Ready Nation.

The survey is designed to evaluate external impressions of the effectiveness of both the products and the communication channels by which the NWS delivers critical, time-sensitive information. This information is often used by emergency managers, state and local first responders, and broadcast media to make weather-related decisions that in turn impact the public. The information is most often communicated via the Web in the form of webinars, briefings, or real-time chat sessions.

2. Explain how this survey was developed. With whom did you consult during the development of this survey on content? statistics? What suggestions did you get about improving the survey?

The survey was developed jointly, including McKinsey leadership with specific expertise on survey methodology as well as subject-matter expertise on organizational behavior and crisis management, OOE, IDSS Regional Program Managers located across the country, and NWS Deputy Director Laura Furglione. In order to maintain objectivity of the survey, the statistical analysis will be supported by the McKinsey OWA team (located at NWS Headquarters in Silver Spring, MD) as well as selected experts within McKinsey’s Survey Development Center.

During this process, the survey team solicited a variety of input. Specific recommendations included:

• Tailoring the self-identified affiliation to ensure that categories are comprehensive of all groups who could be possible recipients of NWS IDSS efforts

• Evaluating specific aspects of effectiveness, including accuracy, timeliness, and relevance

• Tailoring the suite of products surveyed to include the most relevant items that are provided to external partners.

3. Explain how the survey will be conducted. How will the customers be sampled (if fewer than all customers will be surveyed)? What percentage of customers asked to take the survey will respond? What actions are planned to increase the response rate? (Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to services provided by Web.)

The survey will be administered via a Web link sent via on-site Warning Coordination Meteorologists (WCM) directly to their respective stakeholders in the field with whom the NWS already has a significant working relationship. Each WCM will be requested to send the link to approximately ten contacts with whom he/she has regular contact. These contacts will most likely be Emergency Managers, state and local elected officials, members of the media, or other leaders who rely on NWS information in order to make decisions pertinent to public safety. The true number of external stakeholders across the NWS field offices is quite high; however, this referral link will allow NWS to identify those with whom on-site personnel have regular contact. Forwarding the survey as an anonymous link will allow NWS to identify the roles and locations of personnel who regularly use Weather Service information without collecting any PII related to the respondents. Since this survey will be a one-time poll sent, for distribution, to core partners with whom the NWS already has strong working relationships and who receive NWS products via the web, distributing the survey via a web link will be the best way to collect opinions.

Response rates will likely vary by field office, but given that this is an extremely important project to senior leadership at the National Weather Service and that the links will be sent personally by a contact who will be choosing respondents, we anticipate that response rates will be higher than those of the general public. We anticipate that the response rate to this survey could be at or higher than 50%. Any further action to increase response rates would be to ask WCMs to send the survey to more external stakeholders. The initial non-responses would still be part of the overall non-response figure.

4. Describe how the results of this survey will be analyzed and used. If the customer population is sampled, what statistical techniques will be used to generalize the results to the entire customer population? Is this survey intended to measure a GPRA performance measure? (If so, please include an excerpt from the appropriate document.)

Currently, the NWS evaluates its efforts internally after adverse weather events in the form of Service Assessments that are conducted after impactful weather events. The results of the survey will be analyzed and compared to internal appraisals of similar products and communication channels. More broadly, the goal of the survey is not to evaluate each product or channel individually, but rather to measure relative strengths and whether these correspond to strengths identified individually. The NWS will use products and channels identified as strengths both internally and externally to isolate best practices that can be transferred to the rest of the organization. Meanwhile, the NWS will evaluate any differences to understand what drives differences in satisfaction and whether modifications would make these products or communication channels even more effective.

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.

Each WCM (approximately 120) will be requested to send the survey link to approximately ten external stakeholders. Assuming a 50% response rate, the NWS anticipate approximately 600 responses to the survey. This would provide a sufficient sample size for the purposes of identifying relative strengths and weaknesses based on external satisfaction. The external stakeholders surveyed will vary widely, as field offices work with decision-makers based on the jurisdictions relevant to their localities. The most common external stakeholders will include: emergency managers at the federal, state, and local levels, first responders at the state and local levels, business and non-profit leaders, and broadcast media members. As mentioned above, respondents will already have working relationships with the National Weather Service. Accordingly, response rates should be higher than a typical customer service survey issued to the general public. A response rate of approximately 50% is anticipated across all customer segments. This survey has not been conducted before by the National Weather Service and will only be conducted once (non-recurring).

2. Describe the procedures for the collection, including: the statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

The goal of the survey is to evaluate the impressions of external stakeholders of the NWS IDSS efforts. To do so, we will rely on the referrals of individual WCMs located in NWS field offices to identify their most common partners. This referral system will provide a statistical sample that is representative of the population as a whole. No specialized sampling procedures will be required for sufficient accuracy of the survey. As this survey will be a one-off survey, the expected burden on each stakeholder is expected to be minimal for the purposes of this survey.

3. Describe the methods used to maximize response rates and to deal with nonresponse. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield "reliable" data that can be generalized to the universe studied.

Response rates will be maximized because the respondents will be personally identified by a contact within the NWS with whom they have regular contact. Personal referrals tend to have higher response rates than more general inquiries. Non-responses will be identified on a site-by-site basis, as we will identify the office with whom respondents most often receive weather-related IDSS information.

4. Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved OMB must give prior approval.

Not applicable: there will not be any tests of procedures undertaken as part of the survey. The survey will be a one-time information request that will be used to identify the relative strengths and weaknesses among the different products and communication channels that constitute the IDSS efforts of the NWS today.

5. Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The contact at National Weather Service for the survey is Joanne Swanson Kagan, Meteorologist and Principal Advisor, Office of Organizational Excellence. Her contact phone is (301) 427-9066. The survey will be conducted through an external contractor, McKinsey and Company, using a survey link established through McKinsey’s Survey Development Center. The results will be tabulated through McKinsey’s Survey Development Center to ensure that results are not attributed to a specific respondent.