Supplemental and Part B questions

A. Supplemental Questions for DOC/NOAA Customer Survey Clearance
(OMB Control Number 0648-0342)

1. Explain who will be conducting this survey. What program office will be
conducting the survey? What services does this program provide? Who are the customers?
How are these services provided to the customer?
The NOAA Coastal Services Center (CSC) provides a vast array of technical assistance to a variety
of customers, with the primary customer base being state and local coastal managers and planners,
other federal agency partners, as well as non-profit organizations. CSC has defined technical
assistance as “involving direct contact with customers performed by CSC staff (including
contractors), but excluding training events.” 1 The assistance provided by CSC covers a variety of
areas, including facilitation and social science services, assistance with geographic information
systems (GIS) data and mapping, working with coastal-related data maintained by CSC, and
assistance with a variety of software and mapping tools developed and offered by CSC. The
assistance is provided in a variety of formats including: in-person, telephone, email, web conferences
and seminars, and through the CSC web site. CSC provides this assistance free of charge and places
emphasis on timely and accurate responses to the requests made by customers.
2. Explain how this survey was developed. With whom did you consult during the development of
this survey on content? Statistics? What suggestions did you get about improving the survey?
CSC consulted with Eastern Research Group, Inc (ERG) on the development of survey instrument
content, development of sampling approach, and statistical methods. ERG has significant experience
assessing technical assistance provided by Federal agencies through detailed interviews, focus
groups, and surveys that focus on customer satisfaction and outcome attainment.
Under this data collection, CSC intends to have ERG perform a telephone survey to determine the
usefulness of the assistance it has been providing to its customers and customer satisfaction with this
assistance. The survey will collect information on: 1) the extent to which the assistance has been
useful, 2) areas where improvement is needed, 3) the return on investment generated from the
provision of the assistance, and how responses can be used to prioritize the provision of different
types of assistance. Additionally, the survey will provide data that CSC can use to track two of its
internal performance measures: (1) the number and percent of target audiences reporting that CSC
technical assistance activities met their needs and (2) the number and percent of users reporting
satisfaction with the quality of the CSC technical assistance.
CSC contracted with ERG to develop the survey instrument. The survey development process was
informed by reviewing relevant CSC documents and reports, synthesizing CSC documentation of
technical assistance, and interviews with CSC staff and a small set (fewer than nine) of CSC
customers. CSC provided ERG with spreadsheets documenting instances of technical assistance
provided by CSC staff , case studies of recent technical assistance and CSC annual summaries of
technical assistance activities, among other materials to consider in developing an instrument. These
materials provided an overview of the types of technical assistance in scope for the survey as well as
context for interpreting data. . This information provided insights on the population of customers
served by the CSC, the types of requests for assistance they make, and the amount of time CSC staff
devote to providing assistance.
1

The survey will not cover customer satisfaction with CSC products which is collected through other means.

1

ERG performed a set of in-depth interviews with both CSC staff and its customers. ERG performed
telephone and in-person interviews with CSC staff to develop background information and generate
a list of recent technical assistance recipients to interview about their experience. Next, ERG
performed telephone interviews with seven CSC customers to better understand the assistance being
provided from the customers’ perspective; in other words, to understand how the customer thinks
about technical assistance and which aspects of assistance are most important. The insights gained
from these interviews and the materials provided were then used to develop the survey instrument. In
particular, the interviews assisted ERG in developing a set of topics that should be covered by the
survey:
•
•
•
•
•
•

Access: How easy was it for a customer to access CSC?
Needs: Was CSC able to meet the needs of the customers?
Goals and Objectives: How well has CSC assisted the customer in meeting his/her objectives?
Experience: What sort of experience did the customer have in making a request from CSC?
Confidence: How confident are customers in the assistance they received from CSC?
Satisfaction: How satisfied are CSC customers with the services received?

Questions were developed for each category to form the survey instrument. Thus, the interviews
with CSC staff and its customers led to the formulation of both questions and categories (concepts)
under which those questions could be grouped. The data collected under the survey can be analyzed
both at the question level and at the category level. Thus, CSC can track how well it is doing with
regards to access, meeting needs, etc.
3. Explain how the survey will be conducted. How will the customers be sampled (if fewer than
all customers will be surveyed)? What percentage of customers asked to take the survey will
respond? What actions are planned to increase the response rate? (Web-based surveys are not
an acceptable method of sampling a broad population. Web-based surveys must be limited to
services provided by Web.)
CSC will have ERG conduct a telephone survey with customers. CSC chose the telephone survey as
the preferred mode of implementation due to suspected web-based survey fatigue among the target
population. The telephone survey will incorporate a set of close-ended response option questions and
a few open-ended questions. To implement the survey, ERG will perform telephone interviews of
80-100 respondents targeting an on-phone time of no more than 15 minutes.
In order to select the sample of 80-100 respondents for the telephone survey, CSC will develop a list
of telephone numbers of customers who received technical assistance from a CSC staff member
during FY2009 – FY2011. It is expected that those that made contacts to CSC over that time frame
would recall the interaction in enough detail to provide answers to the questions needed for this
analysis; also, the longer time frame provides a larger set of potential respondents from which to
select the sample.
To select the sample, ERG will allocate the sample across four broad categories of assistance:
• Social science and process skills (e.g. needs assessments, survey design, economics,
instructional design, focus groups, project/program design and evaluation, and strategic
planning);
• Technology (e.g. remote sensing, geographic information systems [GIS], and use of CSC
tools);

2

• Data (e.g. determining data standards and data sharing methods, mapping and processing
techniques, and CSC data offerings);
• Stakeholder engagement and collaborative processes (e.g. meeting design including
identifying stakeholders and establishing meeting goals, objectives, and agendas, meeting
facilitation, Participatory GIS, and the Roadmap for Adapting to Coastal Risk).
ERG will allocate the sample equally across the four groups. Equal allocation will allow for having a
sufficient number in each category to perform some analysis on the results at the group level. To
select the sample within each category, ERG will use systematic sampling. The sampling approach is
described in more detail in Section B below.
CSC expects that approximately 80 percent of sampled respondents will complete the survey. First,
many of the respondents work with CSC products and services on a daily or weekly basis and thus
have a vested interest in providing feedback to CSC. Second, in performing the initial interviews for
this effort, ERG contacted nine individuals to set up interviews and was able to complete seven of those
in a timely manner.
Despite the expectation that a high response rate is achievable, CSC and ERG will follow good
survey practices, including the following:
• CSC will send the potential respondents a pre-notification email to inform them of the
upcoming survey.
• For those that refuse at the time of initial contact, ERG will provide the respondent with the
option of scheduling a time that would be more convenient. ERG will also provide the option
of taking an online version of the survey 2 following the second refusal should the respondent
want to take the survey on his/her own time.
• ERG will attempt to make contact with members of the initial sample list up to three times
before selecting replacements.
Not all respondents may be asked the full set of questions. The survey instrument contains a question
that asks the respondents about the importance of their request. Respondents that consider their
request “of little importance” or “not important” are asked only 13 of the remaining 28 questions
following the question on importance of the request, or a total of 19 of the 34 total survey questions. 3
A complete response is considered anyone who answers at least this reduced set of 19 questions. The
purpose of reducing the questions for those that consider their request of lesser importance is to
reduce the burden on those respondents. Nevertheless, the set of 19 questions being asked will be
provide sufficient information to perform analyses. However, we expect that most respondents will
take the full survey. The survey instrument includes two screening questions at the beginning that
are designed to remove those that cannot remember their request to CSC. Presumably, many that
cannot remember their request well also made requests that were “of little importance” or “not
important.”

2

The online version will be identical to the telephone version for most questions. However, the online will provide
the response options for each agreement scale question. For some questions on the telephone survey, response
options are listed on the telephone script, but the interviewer is instructed to not read the option and use the list for
coding purposes. For these questions, the online version will provide the response options and an “other” box.
3
The question on importance of the request is question number 6 on the survey.

3

4. Describe how the results of this survey will be analyzed and used. If the customer
population is sampled, what statistical techniques will be used to generalize the results to
the entire customer population? Is this survey intended to measure a GPRA performance
measure? (If so, please include an excerpt from the appropriate document.)
CSC will use the data collected under this data collection to assess how well it has been providing
technical assistance and to identify areas where improvement may be needed. Additionally, the
survey will provide data that CSC can use to track two of its internal performance measures: (1) the
number and percent of target audiences reporting that CSC technical assistance activities met their
needs and (2) the number and percent of users reporting satisfaction with the quality of the CSC
technical assistance.
Assuming enough variation exists in the data, CSC may have ERG perform a set of multivariate
analyses that link respondents’ answers on outcome attainment and/or satisfaction to their ratings of
CSC technical assistance. The purpose of these analyses will be to assess which ratings are most
closely aligned with CSC customers’ attaining their outcomes and being satisfied with the assistance
provided. If strong linkages between outcome attainment and ratings of technical assistance can be
identified, then CSC could, potentially, track those ratings as indicators of outcome attainment and
satisfaction. This may simplify data collection in the future since fewer data items would need to be
collected.
Although the data does not directly contribute to a GPRA measure, the data from this survey will roll
up into one of NOAA’s GPRA measures: Percentage of U.S. coastal states and territories
demonstrating 20% or more annual improvement in resilience capacity to weather and climate
hazards (%/yr.). NOAA has identified technical assistance as one approach to developing resiliency
capacity. Thus, tracking the extent to which the assistance has been useful and customers satisfaction
with the assistance will contribute the NOAA-level GPRA measure.
The statistical design that will be used to generalize the results to the population is discussed under
Section B, Question 2 below. In short, the sample size being employed will allow for a fair degree of
accuracy for the types of questions being used in the survey (i.e., five point scales) and ERG will
employ weighting to accurately estimate population parameters.
B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
1. Describe (including a numerical estimate) the potential respondent universe and any sampling
or other respondent selection method to be used. Data on the number of entities (e.g.
establishments, State and local governmental units, households, or persons) in the universe and
the corresponding sample are to be provided in tabular form. The tabulation must also include
expected response rates for the collection as a whole. If the collection has been conducted
before, provide the actual response rate achieved.
Target population and sampled populations
For this survey, CSC has defined technical assistance as “involving direct contact with customers
performed by CSC staff (including contractors), but excluding training events.” The purpose of the
assistance is to help organizations and individuals better meet their goals and objectives in relation to
CSC’s areas of expertise. Thus, the target population for this survey is organizations or individuals
that use CSC technical assistance. The sampled population, however, will be individuals (including
those at organizations that work in the coastal management field) for which CSC has managed to
obtain contact information. The sampled population diverges from the target population on the

4

organizational aspect because CSC will be interviewing individuals and asking their opinions and
not asking them to speak for their organizations as a whole. Second, the sampled population is
restricted to just those for which CSC has contact information, in particular telephone numbers. 4
Estimated Universe, Sample Size and Expected Response Rates
Table 1 provides CSC’s estimates of the universe for this survey effort, along with the sample size
and the expected response rates. The estimated universe was derived by totalling data from CSC
internal tracking of assistance for some types of assistance and by using estimates based on
interviews with CSC staff.
Table 1 - Estimated Universe, Sample Sizes, and Response Rates

Category of Technical
Assistance
Social Science and Process
Skills
Stakeholder engagement
and collaborative
processes
Technology
Data
TOTALS

Estimated
Universe
(Population)
[a]

220 [b]
563
750
1,226

Targeted
Sample Size

Expected
Response
Rate

Number of
Contacts
Needed to
Achieve
Target
Sample Size

25

80%

32

25

80%

32

25
25
100

80%
80%
80%

32
32
128

[a] The values in this column were derived from internal tracking documents maintained by CSC as well as interviews with
CSC staff that perform the assistance. The estimates reflect the number of assistance events over a 2.5 year time frame. The
2.5 year time frame was chosen since the survey covers assistance provided in FY09 – FY11. CSC expects that contact
information for approximately half of FY11 will be available when the survey is implemented.
[b] This is the combined amount for social science and process skills and stakeholder engagement and collaborative processes.
The data that CSC compiled at this time for these categories could not be separated. However, CSC is working to develop
new lists for this survey effort that would allow for separating these categories.

2. Describe the procedures for the collection, including: the statistical methodology for
stratification and sample selection; the estimation procedure; the degree of accuracy needed
for the purpose described in the justification; any unusual problems requiring specialized
sampling procedures; and any use of periodic (less frequent than annual) data collection cycles
to reduce burden.
Statistical Method for Stratification and Sample Selection
The sample size for this project was determined based on available budget for the project. However,
as will be discussed under accuracy, the sample size allows for fairly accurate estimates of
parameters for the population of all technical assistance recipients.
The sample is being allocated equally across four strata (see Part A, Question 3 for definitions of the
strata). CSC has determined that the sample should be allocated in this way to ensure that each
stratum has enough sample units to perform meaningful analysis and to assist in identifying strengths
and weaknesses of technical assistance at the category level.

4

CSC is in the process of compiling a phone number list of those that requested technical assistance.

5

CSC has tasked ERG with selecting the sample. To select the sample within each stratum, ERG will
use systematic sampling. To start, ERG will first randomize the list of respondents in each stratum,
then sort each list by the type of assistance received (e.g., GIS modeling, facilitation). Next, ERG
will determine the sampling interval by dividing the total number in each stratum by the sample to
be selected. For example, with a sample of 20 respondents and N names with telephone numbers in
the stratum, the sampling interval would be k = N/20. ERG will then select a random number
between 1 and k which becomes the starting point for the sampling process. The next step would be
to select every kth potential respondent beginning at the randomly selected starting point in the sorted
list. For example, if the random number selected as the start point was 3, then ERG would select
respondent numbers 3, 3 + k, 3 + 2k, etc. Sorting by the in-scope activities within each stratum and
then using systematic sampling will allow in-scope activities to be represented according to their
representation in the list. When replacements are needed due to refusals, ERG will use the next
respondent on the list to replace the refusal. For example, if respondent number 3 + k were to refuse
to participate (or not be available), then ERG would replace that respondent with respondent number
3 + k + 1 (i.e., the respondent that follows 3 + k).
Estimation Procedure
Population parameters will be estimated by appropriately weighting the sample responses. The same
number of sample units is being drawn from each stratum. Thus, each stratum’s sample represents a
different proportion of the stratum’s population. The weight that will be used will reflect the inverted
probability of selection into the sample. These are calculated as follows:

where h represents the strata (i.e., h = 1, 2, 3, or 4) and wh is the weight for the hth stratum, Nh is the
population for the hth stratum, and nh is the sample selected from the hth stratum. These weights will
be multiplied by respondent values for each question to calculate population parameters such as
means and totals and in calculating population variances.
Accuracy
The sample size for this data collection will allow ERG to estimate population parameters with a fair
degree of accuracy. The main questions on this survey use five point scales which, for analysis, will
be converted to numerical values ranging from 1 to 5. Based on the sample size, ERG can be fairly
certain that the population value for the five point scale will be within a half-point of the sample
value. This is based on a 90 percent confidence interval and 80 percent power to detect a significant
difference. This is based on Cohen’s (1988) 5 sample size calculation for two-sided 90 percent
confidence intervals with 80 percent power. Cohen’s formula for sample size (n) in this situation is 6

where the value 1,237 is derived from Table 2.4.1 in his book and

5
6

Jacob Cohen, 1988. Statistical power analysis for the behavioral sciences, Lawrence Erlbaum Associates.
This corresponds to Cohen’s case 3 in Chapter 2 of this book.

6

where D is the required precision and s is the standard deviation. Our concern here is the level of
precision (D) that can be obtained for a sample of 80-100 respondents. Thus, we solve the sample
size equation (n) for D and then substitute in values of 80 and 100 for n. The other unknown, s, is
assumed to be the square root of 1.2. 7 Performing that calculation, we find that a sample size of 80 is
associated with a precision of 0.43 points on the five point scale and a sample size of 100 is
associated with a precision of 0.39 on a five point scale. Thus, to be conservative, we can state that
the sample of 80-100 respondents will allow ERG’s sample-based estimates to be within 0.5 of the
population values on a five point scale with 90 percent confidence.
Unusual Problems Requiring Specialized Sampling Procedures
None are required.
Use of Periodic (Less Than Annual) Data Collection
This request is for a one-time data collection. CSC may, in the future, collect these data on an annual
basis.
3. Describe the methods used to maximize response rates and to deal with nonresponse. The
accuracy and reliability of the information collected must be shown to be adequate for the
intended uses. For collections based on sampling, a special justification must be provided if
they will not yield "reliable" data that can be generalized to the universe studied.
Despite the expectation that a high response rate is achievable, CSC and ERG will follow good
survey practices, including the following:
• CSC will send the potential respondents a pre-notification email to inform them of the
upcoming survey.
• For those that refuse at the time of initial contact, ERG will provide the respondent with the
option of scheduling a time that would be more convenient. ERG will also provide the option of
taking an online version of the survey 8 following the second refusal should the respondent want
to take the survey on his/her own time.
• ERG will attempt to make contact with members of the initial sample list up to three times
before selecting replacements.
To replace refusals, ERG will develop a replacement list that maintains the systematic sampling
properties. ERG will select replacements from this list. When replacements are needed due to
refusals, ERG will use the next respondent on the list to replace the refusal.
As discussed above (Section B, Question 2, Accuracy), the sample size for this data collection
should allow for a fair degree of accuracy in estimating population parameters. CSC expects that
being within a half-point of the population parameters with 90 percent confidence and 80 percent

The value of 1.2 is taken from Edward F. McQuarrie, 2006, The market research toolbox: a concise guide for
beginners, (Sage Publications). McQuarrie notes that a good estimate of the variance of a five point scale is 1.2.
8
The online version will be identical to the telephone version for most questions. However, the online will provide
the response options for each agreement scale question. For some questions on the telephone survey, response
options are listed on the telephone script, but the interviewer is instructed to not read the option and use the list for
coding purposes. For these questions, the online version will provide the response options and an “other” box.
7

7

power will be more than sufficient to generate reliable results that CSC can use to assess its
provision of technical assistance.
4. Describe any tests of procedures or methods to be undertaken. Tests are encouraged as
effective means to refine collections, but if ten or more test respondents are involved OMB
must give prior approval.
CSC contracted with ERG to develop the survey instrument. ERG has developed a number of
surveys related to assessing customer satisfaction with assistance. Additionally, as part of ERG’s
work, ERG performed 17 interviews with CSC staff and performed seven interviews with CSC
customers. Those interviews were used to develop and refine the survey instrument being used in
this survey.
5. Provide the name and telephone number of individuals consulted on the statistical aspects of
the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who
will actually collect and/or analyze the information for the agency.
CSC has contracted with Eastern Research Group, Inc. (ERG) of Lexington, MA to design the
survey instrument, develop the sampling approach, implement the survey, and analyze the resulting
data collected. ERG’s project manager for this work is Dr. Lou Nadeau (781-674-7316;
[email protected]).

8