Supplemental and Part B Questions

Supplemental Questions for DOC/NOAA Customer Survey Clearance

(OMB Control Number 0648-0342)

1. Explain who will be conducting this survey. What program office will be conducting the survey? What services does this program provide? Who are the customers? How are these services provided to the customer?

The proposed information collection will be led by Dr. Vankita Brown of the National Weather Service (NWS), supported by an interdisciplinary team of social scientists representing various NOAA line offices, other federal agencies and academic institutions. The program office responsible for the proposed information collection is the NWS Office of Climate Water, and Weather Services.

The NWS conducts Service Assessments (both nationally and regionally focused) to evaluate its performance after significant hydrometeorological, oceanographic, or geological events. NWS also collects routine feedback around other events that generally lead to more localized impacts (e.g, dense fog, frost/freeze, high surf, etc.). Service Assessments may be initiated when one or more of the following criteria are met:

• Major economic impact on a large area or population

• Multiple fatalities or numerous serious injuries

• Extensive national public interest or media coverage

• Unusual level of attention to NWS performance

Assessment teams composed of experts from within and outside the NWS, evaluate activities before, during, and after events to determine the usefulness of NWS products and services. The team generates a report, which serves as an evaluative tool to identify and share best practices in operations and procedures, identify and address service deficiencies, and make recommendations to revise and/or develop new policies The goal of the activity is for the NWS to continuously improve its services to the nation.

The NWS Office of Climate, Water and Weather Services (OCWWS) oversees delivery of hydrology, meteorology and climate services and designs and implements new products and services in numerous service areas. These service areas include:

• Aviation Weather

• Climate

• Fire Weather

• Hydrology

• Marine Weather

• Outreach, Education and Verification

• Observations

• Operations and Change Management

• Public Weather

• Service Change Notifications

• Training

More specifically, OCWWS does the following:

• Conducts customer and partner outreach

• Validates and document service requirements

• Develops, coordinates, and issues national service and operations policy

• Develop long-term service improvement strategies and proposals

• Develops, implements and oversees service delivery performance measures

• Evaluates operational service efficiency, effectiveness, and customer satisfaction

• Develops NWS policy delineating NWS' mission and its public sector responsibility to produce and disseminate information and the resulting public-private sector roles

These activities require collaboration with the climate and hydrometeorological communities as well as federal, state, and local government agencies; the media; the private weather information sector; universities; non-governmental organizations; constituent organizations; and most important with regard to this proposed information collection, the public.

2. Explain how this survey was developed. With whom did you consult during the development of this survey on content? Statistics? What suggestions did you get about improving the survey?

The proposed research was planned in close consultation with a team of NWS meteorologists with extensive expertise in the full array of weather event types (corresponding to survey instruments), as well as NWS headquarters staff, and social science staff from the NOAA Coastal Services Center (NOAA/NOS). A high level of effort was expended to draft a collection of surveys that is consistent across all 18 weather event types to allow for meaningful comparisons across respondents. The proposed survey methodology was pilot tested and evaluated for elements including question comprehension, readability and relative ease of completion, and estimated response time. Additionally, pilot participants were administered paired instruments to elicit feedback on how question interpretation may differ when considering different weather event types (for example questions within the context of tornados versus the same questions focused on wind chill/extreme cold. The survey questions for the proposed study were developed through reviewing the list of cleared questions from the NOAA Generic Clearance, OMB Control No. 0648-0342. The NWS survey team identified areas in which the survey questions possessed certain limitations in question clarity, order, variable measurement, and ability for participants to meaningfully respond to question items. Thoughtful effort has been made to keep the proposed surveys as brief as possible, inquire only upon tangible actions, and to ensure relevance to the primary weather event types.

Further, the NOAA Coastal Services Center’s Human Dimensions Program possesses extensive expertise in the areas of customer interviews, survey design and administration, focus group administration, and product evaluation and has closely collaborated with NWS in the development and piloting of the proposed surveys.

3. Explain how the survey will be conducted. How will the customers be sampled (if fewer than all customers will be surveyed)? What percentage of customers asked to take the survey will respond? What actions are planned to increase the response rate? (Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to services provided by Web.)

The information collection will be administered via a guided, intercept survey (i.e., a survey administrator will obtain informed consent and then guide the participant through a paper version of the survey). Since this work is based upon evaluating public satisfaction with NWS’ performance in providing relevant and meaningful information following significant weather events, surveys will be administered as immediately as possible, in discreet geographies where weather events have taken place. For Service Assessments, such as the assessment for Superstorm Sandy, a Department of Commerce Administrator can call for such an assessment to take place. For other events, where there is less significant, national interest, the Director of the NWS orders the survey to take place. It should be mentioned that the proposed survey is only a portion of the overall service assessment effort, which includes a technical assessment of NWS Weather Forecast Office (WFO) deployed products, and hardware (RADAR, etc.); and discussions with broadcast meteorologists, emergency managers, elected officials, and transportation and aviation authorities.

The target audiences for Service Assessments and routine information collection surveys are the American public and core partners such as emergency managers and broadcast meteorologists Sampling is particularly challenging for such collections. In many instances, following extreme weather events, members of a community are very busy with tidying up their lawns, making home repairs, and in extreme cases, displaced from their homes or being housed temporarily in emergency shelters and community centers. The research team will closely coordinate with local-level emergency managers and NWS WFO staff to determine the most effective means to access potential respondents for each collection. These two entities are the most connected to the local communities in terms of being aware of where emergency shelters have been designated and the of most effective means for accessing its citizens, post-event. A random sample of community members will be solicited for this information collection. If citizens have been displaced from their homes, sampling and information collection will take place at community centers and emergency shelters. If conditions are less severe and residents remain at home, them a random sample of households will be implemented.

For the proposed collection, it will be possible to calculate an actual response rate for reporting purposes. The number of questions has been made as brief as possible in an effort to maximize response and participation. The intended sample size is 100 individuals per collection. Based on previous NWS Service Assessments, the response rate is expected to be relatively high (approximately 80%), resulting in 80 completed surveys per collection. It is estimated that approximately two NWS Service Assessments will be required each calendar year, and approximately four routine information collections for more localized events (dust storms, frost/freeze events, flash flood, dense fog, etc.). The specific surveys administered each year are completely dependent on weather event types as they occur.

4. Describe how the results of this survey will be analyzed and used. If the customer population is sampled, what statistical techniques will be used to generalize the results to the entire customer population? Is this survey intended to measure a GPRA performance measure? (If so, please include an excerpt from the appropriate document.)

Analysis of survey data will be undertaken through basic descriptive statistics only (e.g., percent, mean scores). This information collection seeks to assess the degree to which NWS effectively communicated weather-related risk to the public. The respondent universe will include a sampling of the general public. This general public audience is large and somewhat undefined, particularly after a severe weather event. The research team will rely heavily on NWS Weather Forecast Office and local emergency manager contact within their regions to aid in identifying individuals and their locations. The sample will be representative of members of the community/region, but not necessarily representative of all Americans. This will be clearly articulated in subsequent reporting of the results.

PART B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.

The respondent universe for the proposed service assessment is a sampling of the general public within discreet geographies that were impacted by one, or an assemblage of the 18 weather events of interest (these 18 weather event types correspond with the accompanying survey instruments).

Following is the breakdown of the population of interest.

Population Anticipate an 80% response rate	% of Participants (total 100%) 600 individuals solicited (100 people x 6 collections)
General Public (households)	100% (target: 600, estimated responses:480)

The estimated time necessary for each respondent to complete the survey is 7 minutes, based on trials with a small pilot sample. Total estimated public burden associated with this information collection is 56 hours (480 surveys @ 7 minutes per response).

2. Describe the procedures for the collection, including: the statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

The information collection will be administered via a guided, intercept survey (i.e., the survey administrator will obtain informed consent and then guide the participant through a paper version of the survey). Questions will inquire (post event) upon the effectiveness of information conveyed to the public by the NWS leading up to adverse weather events.

A random sample of the public will be instituted for this information collection. It is possible that residents will be displaced from their homes due to weather emergencies. If so, researchers will coordinate with local emergency managers and NWS WFOs to locate residents of the area at designated emergency shelters and community centers. These two entities are the most connected to the local communities in terms of being aware of where emergency shelters have been designated and the of most effective means for accessing its citizens, post-event. While the small N will likely not ensure an entirely representative sample, steps are being taken to ensure the sample is as representative as possible, such as the inclusion of demographic questions to account for representation on a number of variables. Additionally, it is not the intent of such research to make sweeping generalizations across the entire U.S. public, but rather, to obtain targeted feedback within discreet communities or regions to improve NWS service offerings. Survey data will be stored on a password-protected work space at NWS headquarters, accessible only by staff particular to this project. Respondents will be instructed not to provide identifying information in their interviews (names, social security numbers, dates of birth, etc.), and any identifying information provided will not be recorded by survey administrators.

3. Describe the methods used to maximize response rates and to deal with nonresponse. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield "reliable" data that can be generalized to the universe studied.

The intent of this information collection is to assess public feedback on NWS services and communications. The intended approach will yield an informed sample of the respondent universe – and the information gained will be extremely valuable in making severe weather communication improvements. This survey sampling process for this collection request is challenging, but the proposed plan provides the best opportunity for the target population to provide feedback on NWS Service Assessments and routine feedback related to NWS customer satisfaction.

In order to improve response rates for this information collection, the survey process has been made as concise as possible. Additionally, the research team will attempt to alleviate potential nonresponse bias by attending closely to demographic responses within the survey (age, gender, race, education level, household income).

4. Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved OMB must give prior approval.

Pilot surveys were administered to, and received comments from, fewer than ten representative members of the target population. Reviewers were asked to offer feedback on the length, appropriateness and clarity of questions, content, or other aspects to improve the interview process. Comments from reviewers were quite helpful and resulted in design, and content changes to clarify questions and simplify instructions.

5. Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The primary point of contact for this information collection request is Dr. Chris Ellis at the NOAA Coastal Services Center, available by telephone at (843) 740-1195 or by email at [email protected]. It is being sent on behalf of the National Weather Service, Office of Climate, Water, and Weather Services.