Supplemental and Part B Questions

1. Supplemental Questions for DOC/NOAA Customer Survey Clearance
   (OMB Control Number 0648-0342)

1. Explain who will be conducting this survey. What program office will be
   conducting the survey? What services does this program provide? Who are the customers? How are these services provided to the customer?

The Coastal Storms Program (CSP) is a nationwide effort led by the National Oceanic and Atmospheric Administration (NOAA) Coastal Services Center (CSC) to make communities safer by reducing the loss of life and negative impacts caused by coastal storms. This work is accomplished by bringing together Federal, state, and local governments, non-profits, and other stakeholders. NOAA sets up and runs a program in a specific region for approximately five years and then moves on to other regions. The idea is that after five years, the program will have set up the infrastructure for the communities in the region to continue on its own. When the program is active in a region, a local outreach coordinator is provided to oversee the effort. The results often include new data and predictive tools, new ways of keeping the public informed and enlightened about coastal storm preparedness, and new partnerships that strengthen existing resilience efforts. The program attempts to increase resiliency in the project area through outreach, product development, and provision of small grants to local stakeholders to fund projects to increase resiliency.

The program has funded projects in Northeast Florida, the Gulf of Mexico, Southern California, and the Pacific Northwest with actively funded projects for fiscal year (FY) 2012 in the Pacific Islands and the Great Lakes. Each funded project lasts three to five years and brings additional manpower, focus, and funding to a specific region.

The NOAA CSC is seeking approval to collect information on how well the CSP has met the needs of stakeholders/customers in the Gulf of Mexico, the extent to which the program’s products and services has satisfied those customers, and ways to improve the program in the future. The activities that the CSP performed in the Gulf are similar to the activities that the program uses in other regions. CSC is seeking approval on three data collection efforts to assess how well the program has met the needs of its customers:

• A survey of program stakeholders in the Gulf of Mexico. This survey of approximately 220 stakeholders will collect information on awareness, use, and usefulness of program products and services as well as overall satisfaction with the program. The survey will also collect information on a number of mid- and short-term outcomes targeted by CSC under the program related to increasing resiliency and awareness of issues.

• Interviews with small grantees. CSP also provided small grants to local stakeholders to develop tools and services related to increasing resiliency in the region. These interviews will ask grantees about their grants, the grant process, and the extent to which the outputs from their grants were useful to customers in increasing resiliency in the region. We expect to interview about 20 individuals with these interviews.

• Interviews with communities that participated in the Community Resiliency Index (CRI) sessions. One of the key products of the program in the Gulf was the Community Resiliency Index, a tool that helps communities assess their risk from coastal disasters and approaches for reducing that risk. The CRI was implemented in a set of 35 communities using facilitated sessions, and CSC will interview 1-2 individuals in each of these communities. These interviews will focus on the communities’ view of the process, how and why the process met their needs, and improvements in resiliency generated by the CRI process.

2. Explain how this survey was developed. With whom did you consult regarding content during the development of this survey? Statistics? What suggestions did you get about improving the survey?

CSC consulted with Eastern Research Group, Inc. (ERG) on the development of the survey instrument and interview guides as well as the organization of the data collection (i.e., using three distinct data collections). ERG has significant experience assessing technical assistance provided by Federal agencies through detailed interviews, focus groups, and surveys that focus on customer satisfaction and outcome attainment.

CSC contracted with ERG to develop the survey instrument and interview guides. The survey development process was informed by reviewing relevant CSP documents and reports, synthesizing CSC documentation of technical assistance, and conducting interviews with 12 CSP program staff and a small set (fewer than nine) of CSP stakeholders in the Gulf.

As noted, ERG performed a set of in-depth interviews with both CSP program staff and its customers. ERG performed these interviews to better understand the program and how it works. Based on these interviews, ERG determined that three data collections were needed. The first form of data collection is the survey, which covers all stakeholders and asks about: (1) awareness and use of key products and services, and (2) how the program has improved resiliency. The second form of data collection is the interviews with small grantees. Interviewees who spoke with ERG indicated that the small grants portion of the program generated some important work, and the best approach to collecting information on how those grants benefitted customers would be to talk with the grantees themselves. The third form of data collection is the interviews with the communities that participated in the Community Resiliency Index (CRI) sessions. Almost all interviewees who spoke with ERG indicated the importance of the CRI to the program and its potential as an effective planning tool for communities. Thus, ERG recommended that CSC collect data specific to the CRI.

3. Explain how the survey will be conducted. How will the customers be sampled (if fewer than all customers will be surveyed)? What percentage of customers asked to take the survey will respond? What actions are planned to increase the response rate? (Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to services provided by Web.)

Survey of Stakeholders

CSP in the Gulf of Mexico maintained a list of stakeholders and their contact information. CSC will perform the survey using a web-based survey instrument. CSC has chosen a web-based instrument for two reasons: first, many of the tools that we are asking about are web-based, and some require the respondent to see a visual representation of the tool to prompt their memory; second, the CSP has significant interaction with stakeholders over the internet (e.g., via email, in web forums, previous surveys, etc.); therefore, a web-based survey represents a logical mode to implement this survey.

No statistical methods are being used in the participant selection or distribution of the survey; all individuals in the stakeholder listing will receive a survey. CSC expects that between 80 and 90 percent of the recipients will respond to the survey, as these tend to be individuals with whom CSC has worked with since 2007 and with whom CSC maintains regular contact.

Despite the anticipation of a high response rate, CSC and ERG will follow good survey practices, including the following:

• CSC will send the potential respondents a pre-notification email to inform them of the upcoming survey.

• CSC will send the email with the survey link 3-4 days after the pre-notification email.

• CSC will send two reminders to non-responders one and two weeks following the email with the survey link.

Interviews with Small Grantees

CSP maintains a list of its grantees and expects to be able to interview at least one individual associated with each grant. Since most of the grants involve more than one grantee, CSC will interview as many of the grantees per grant as possible. CSC will have ERG contact the grantees to schedule and then perform these interviews. ERG will generate detailed notes from the interviews.

Interviews with CRI Communities

CSC will contact all of the communities that participated in the CRI sessions and schedule interviews with relevant community participants to be conducted by ERG. CSC expects that almost all (90 percent plus) will schedule an interview with ERG. ERG will perform these interviews over the phone, and detailed notes from the interviews will be generated.

4. Describe how the results of this survey will be analyzed and used. If the customer population is sampled, what statistical techniques will be used to generalize the results to the entire customer population? Is this survey intended to measure a GPRA performance measure? (If so, please include an excerpt from the appropriate document.)

CSC will use the data resulting from this data collection to assess: (1) how well the CSP has operated in the Gulf of Mexico, and (2) where improvement may be needed when implementing CSP in other project areas. Additionally, the survey will assist CSC in tracking outcomes related to the program. CSC will have ERG perform thematic analyses on the detailed interview notes to identify key terms, concepts, and trends and relate these findings to the results of other analyses to provide context.

Assuming enough variation exists in the data, CSC may have ERG perform a set of multivariate analyses that link respondents’ answers on outcome attainment and/or satisfaction to awareness and use of CSP products and services.

Although the data does not directly contribute to a GPRA measure, the data from this survey will roll up into one of NOAA’s GPRA measures: Percentage of U.S. coastal states and territories demonstrating 20% or more annual improvement in resilience capacity to weather and climate hazards (%/yr.).

CSC is not using any statistical methods to select participants from the population and will select all population members in the survey and the two sets of interviews.

2. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.

Survey of Stakeholders

The potential respondent universe includes at least 220 individuals who are listed as stakeholders of the CSP in the Gulf of Mexico. The survey will not sample the population, no statistical methods are being used in the participant selection or distribution of the survey; all individuals in the stakeholder listing will receive a survey. CSC expects that between 80 and 90 percent of the recipients will respond to the survey, as these tend to be individuals with whom CSC has worked with since 2007 and which CSC maintains regular contact.

Interviews with Small Grantees

The potential respondent universe includes at least 20 individuals (i.e., one per grant) who received small grants under the CSP in the Gulf of Mexico. No statistical methods are being used in the participant selection or distribution of the survey; all individuals in the listing of small grantees will be contacted for an interview. CSC expects that at least one representative of each grant will respond to the request for interview, as these tend to be individuals with whom CSC has worked with between 2007 and 2012 and with whom CSC maintains regular contact.

Interviews with CRI Communities

The potential respondent universe includes at least 35 individuals who represent communities who participated in the Community Resiliency Index facilitated sessions offered by the CSP in the Gulf of Mexico. No statistical methods are being used in the participant selection or distribution of the survey; all individuals in the listing of CRI communities will be contacted for an interview. CSC expects that at least one representative of each community (usually city planners) will respond to the request for interview, as these tend to be individuals with whom CSC worked interactively with in setting up and conducting the facilitated sessions and with whom CSC maintains regular contact.

2. Describe the procedures for the collection, including: the statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

The answers below apply to all three data collections.

Statistical Method for Stratification and Sample Selection

CSC is not selecting a sample, in each data collection, CSC will seek responses (either through survey or interviews) from the entire relevant population.

Estimation Procedure and Accuracy

In each data collection, CSC does not need to extrapolate the results to the population and will therefore not need to estimate population parameters from the collected data. This also means that the accuracy of the estimates in not meaningful to calculate

Unusual Problems Requiring Specialized Sampling Procedures

None are required.

Periodic Data Collection Cycles

This request is for a one-time data collection.

3. Describe the methods used to maximize response rates and to deal with nonresponse. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield "reliable" data that can be generalized to the universe studied.

Survey of Stakeholders

CSC expects that a high response rate is achievable for the web-based survey: however, CSC and ERG will continue to follow good survey practices to ensure high participation, including the following:

• ERG will send the potential respondents a pre-notification email to inform them of the upcoming survey.

• ERG will send the email with the survey link 3-4 days after the pre-notification email.

• ERG will send 2 reminders to non-responders, one and two weeks following the email with the survey link.

As discussed above, ERG, on CSC’s behalf, will administer the survey to the full population of stakeholders; no statistical methods will be used.

The CSP maintains a database of these respondents and CSC will also be able to assess whether nonresponse is concentrated among certain types of respondents (e.g., local planners, specific areas, etc.).

Interviews with Small Grantees

CSC will contact all of the small grantees to introduce the project, ERG’s role in the project, and request that the small grantees schedule an interview to be conducted by ERG. ERG will follow up with potential interviewees to determine the most convenient time for the interview and answer any questions the interviewees may have about the interview. CSC expects that almost all (90 percent plus) will schedule an interview with ERG.

Interviews with CRI Communities

CSC will contact all of the communities that participated in the CRI sessions to introduce the project, ERG’s role in the project, and request that relevant community participants schedule interviews to be conducted by ERG. ERG will follow up with potential interviewees to determine the most convenient time for the interview and answer any questions the interviewees may have about the interview. CSC expects that almost all (90 percent plus) will schedule an interview with ERG.

4. Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved OMB must give prior approval.

CSC consulted with Eastern Research Group, Inc. (ERG) on the development of the survey instrument. ERG has significant experience assessing technical assistance provided by Federal agencies through detailed interviews, focus groups, stakeholder engagement, and surveys that focus on customer satisfaction with services. The survey development process was informed by reviewing the history and development of the program and through interviews with CSP program staff and a small (nine or fewer) set of stakeholders.

5. Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

CSC has contracted with Eastern Research Group, Inc (ERG) of Lexington, MA to design the survey instrument and interview guides, implement the data collections, and analyse the resulting data. ERG’s project manager for this work is Lou Nadeau (781-674-7316; [email protected]).

Coastal Storms Program: An Evaluation of Outcomes in Gulf of Mexico 7