Supplementary and Part B questions

1
OMB Control No. 0648-0342
Expires 12/31/2011

Summary
This document provides the details of the 2011 Nautical Charting Customer Satisfaction
Survey, which has been updated from the 2009/2010 survey approved under OMB Control No.
0648-0342.
1.

Explain who will be conducting this survey. What program office will be conducting
the survey? What services does this program provide? Who are the customers? How
are these services provided to the customer?

The survey will be conducted by the National Ocean Service’s Office of Coast Survey (OCS).
OCS produces nautical charts in paper and electronic format, the U.S. Coast Pilot, a companion
publication to nautical charts, and other products supporting marine navigation. Nautical charts
provide navigational coverage of ports and harbors and United States territorial waters. These
nautical products are used by commercial vessels and recreational boaters transiting United
States waters and making calls on seaports. Nautical products are provided to customers
through a government warehouse, a network of approximately 1,800 nautical sales agents, and
through the Internet.
This survey is in accordance with Executive Order 12862, the National Performance Review
and good management practices. NOS is seeking to continue to determine whether its
customers/users are satisfied with the services/products they receive and whether they have
suggestions as to how the services/products may be improved in the future or made more useful.
Therefore the information will be used to obtain guidance on how to improve NOS’s products
and services.
The current NOAA strategy focuses on traditional and non-traditional users of NOAA’s
products and services. The 2011 survey will be focused on the largest group of traditional
users, who are commercial mariners, to get to know their needs better and obtain more feedback
about commercial craft sailing, and also the emerging non-traditional potential customers/users
such as those in Coastal Management in Federal, State and commercial business.
2. Explain how this survey was developed. With whom did you consult during the
development of this survey on content? Statistics? What suggestions did you get about
improving the survey?
The 2011 commercial mariner’s survey questions are based on the questions used in previous
surveys, to ensure that there is trend analysis developed. The Coastal Managers section of the
survey has been developed in consultation with the NOAA Coastal Services Center and user
groups at workshop sessions at conventions. Based upon user comments, the length of the
survey for the commercial mariner’s survey has again been reduced compared to past surveys,
which should increase response, but still provide actionable information.

2
In the 2011 survey the commercial mariners and coastal managers will only answer direct
questions, so there will not be any visual representation of charts to view and provide feedback
on, as in 2009. The survey will be administered on line. This on-line version of the
questionnaire will be able to be filled out only by the recipient of the email and he/she will be
able to fill out the survey only once.
Professional assistance in question development has been provided by the survey firm, Strategy,
Research, and Action, who have contributed to producing questionnaires which have been
effective amongst chart users in the past. The 2011 survey is designed and will be administered
using the knowledge gained from previous surveys, which will ensure that the questionnaire
achieves the best balance between maximizing data quality and controlling measurement error,
while minimizing respondent burden and cost.
3. Explain how the survey will be conducted. How will the customers be sampled (if
fewer than all customers will be surveyed)? What percentage of customers asked to
take the survey will respond? What actions are planned to increase the response rate?
(Web-based surveys are not an acceptable method of sampling a broad population.
Web-based surveys must be limited to services provided by Web.)
The universe for the sample is defined as ‘users of charts and nautical products relating to
navigating in US charted waters’. The selection of the sample frame for this survey is based on
many years of reviewing Chart users’ lists in relation to how the data is to be used, to develop
guidance on how to improve services/products.
Completing a census of users is beyond the budget allocation that is available for the
completion of this survey. Therefore NOS has worked with Strategy, Research & Action to
develop a sample to give as representative and unbiased data about chart users as possible.
The aim of this survey is to select a representative sample using a stratified sampling technique,
in which the chosen sample is forced to contain potential respondents from each of the key
segments of the population. Such a sample provides statistical efficiency, as the sampling error
is likely to be smaller than using a systematic random sample.
NOS will send self-completion survey forms to 6,000 commercial mariners and 4,400 coastal
managers with a link to the research company’s server, so that the survey can be completed
online. This is possible because we have lists of product users’ emails.
The Commercial mariners email will mainly be to subscribers to Professional Mariner, as they
have the most up to date list of chart users. The contact lists below are selected to cover all parts
of the population of chart users. The Coastal Managers email will be to those listed by the
[email protected]., which is coordinated and administered by the Delaware Sea Grant
College Program at the University of Delaware.
We show below if each list will be used in its entirety or sampled, and the expected number to
be sampled:

3
1.

Professional Mariner – a sample of 6,000 subscribers from the total of their mailing list
who are commercial mariners

2.

[email protected] – the full list of 4,400 named coastal managers on this list
administered by the University of Delaware.

A response rate of up to 25% for these surveys is likely to be achieved, based on previous
results and because of the following:
•

Accuracy of the Email Lists
NOS has spent time ensuring the accuracy of the email lists and the extent to which
They are segmented into the relevant users of NOS charts. This includes those who are
regularly interacting with NOS for all of its services. Use of these commercial mariners’
lists in the past has also contributed to the high response of 25% for the surveys.

•

Re-contacting the Target Audience
Past surveys have taken into account the recent requirement referenced in all Data
Protection and Codes of Conduct for research*, asking the respondents if they would be
willing to take part in future surveys. A high response of 50% has been achieved to this
question in the past, indicating that NOS had contacted the correct target audience and
this helps to keep the response rate high in future surveys, although it is recognised that
not all those responding affirmatively will take part in the future.

•

The relationship between NOS and its Chart Users
The online self-completion survey has been selected for this survey, because of the
relationship between NOS and Chart users.
Past surveys have shown that the respondents:
o

Are interested in responding, because the survey was sent by NOS

o

Are interested in the subject

o

Believe that taking part in the survey will improve the relationship with, and
service received from, NOS in the future.

*The Standards are administered for International Research by the Market Research Society (MRS) out of the UK
and MRS members are required to use them for projects completed by them internationally.
Standards: Code of conduct
Standards: Process

4

•

•

The Cover email
The questionnaires will be emailed with a cover email signed by a senior member of
NOS staff, which will identify:
o

The purpose of the research

o

The reasons why NOS would like the recipients to respond

o

The ease of completion

o

The short time needed to complete the questionnaire

o

A reminder that no confidentiality can be assured as a result of participating in the
survey, but that respondents can withhold the details of their name and address

o

A NOS and research supplier contact name for more information or questions
about the survey

o

The time scale for return of the questionnaire.

Delivering the NOS Email
In planning this survey, we have done our best to ensure that there is a high response by
taking the following steps:
o Issue an initial email to all the targeted audiences to inform them that a voluntary
questionnaire will be emailed shortly and to raise their awareness of its pending
arrival.
o Email the questionnaire 7 days later. NOS will inform respondents that the
responses to the collection of information are voluntary. The information will
also state that the questionnaire will take no longer than 10 minutes to complete.
o The OMB Control Number and expiration date (OMB Control No. 0648-0342 –
Expires 12/31/2011) and a Paperwork Reduction Act Statement will be shown at the
end of the on-line questionnaire.
o Email a reminder 7 days after the questionnaire has been emailed reminding the
target respondent to return the questionnaire as soon as possible.

Each survey questionnaire will be checked by the research contractor upon receipt. Any
inaccurate completions will be discarded and only questionnaires which have been completed
accurately and with due consideration will be included in the final analysis.
NOS will implement safeguards throughout the survey production process by asking the
research contractors to ensure that survey data are handled to avoid disclosure. This is done by

5
separating the survey responses from classification information, which might identify who has
provided the response.
Most common apparent reasons for non-response in these surveys are refusals or nonavailability of respondents, likely resulting from the timing of the survey; NOS management is
taking care to schedule the survey during periods not including national holidays. Mainly
because we have not had obviously skewed response profiles in past surveys, we did not build
into the survey, or budget for, any follow-up activities with non-responders.
4. Describe how the results of this survey will be analyzed and used. If the customer
population is sampled, what statistical techniques will be used to generalize the results
to the entire customer population? Is this survey intended to measure a GPRA
performance measure? (If so, please include an excerpt from the appropriate
document.)
Analysis and Report Planning
A multivariate analysis will be conducted on the responses and Chart user types and other
selected classification information, such as affiliations, type of Commercial Mariner and type of
Coastal Manager, etc. Means and standard deviations and standard errors for user representation
will also be calculated. The survey is not intended to measure a GPRA performance measure.
These analyses will be comparable with those of previous surveys, so that similar statistical
conclusions can be derived from past survey data as is acceptable statistical practice.
Review of Information Products
NOS will be responsible for the quality of information that they disseminate and will institute
appropriate content/subject matter, statistical and methodological review procedures to comply
with OMB and agency Information Quality Guidelines.
Releasing Information
NOS will produce a research summary which can be used to release information intended for
the general public in line with NOS dissemination policies and procedures. An Appendix will
describe the procedures that have been used to evaluate the quality of the data, to allow users to
interpret results of analyses and to help designers of recurring surveys to focus on improvement
efforts.
Data Protection and Disclosure Avoidance for Dissemination
NOS will implement safeguards throughout the survey production process by asking the
research contractors to ensure that survey data are handled to avoid disclosure. NOS will
ensure that the information that is published or released on request will comply with applicable
Federal legislation and regulations.

6
Survey Documentation
NOS will produce survey documentation that includes those materials needed to analyze data
from the survey, as well as the information necessary to replicate and evaluate the survey’s
results compared with previous surveys.
Documentation and Release of Public-Use Microdata
If NOS releases microdata to the public it will ensure that it will include documentation clearly
describing how the information is constructed and if required will also provide the metadata
necessary for users to access and manipulate the data.
B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
1. Describe (including a numerical estimate) the potential respondent universe and any
sampling or other respondent selection method to be used. Data on the number of
entities (e.g. establishments, State and local governmental units, households, or
persons) in the universe and the corresponding sample are to be provided in tabular
form. The tabulation must also include expected response rates for the collection as a
whole. If the collection has been conducted before, provide the actual response rate
achieved.
The universe for the sample is defined as ‘users of charts and nautical products relating to
navigating in US charted waters’. The aim of this survey is to select a representative sample
using a stratified sampling technique, in which the chosen sample is forced to contain potential
respondents from each of the key segments of the population. Such a sample provides
statistical efficiency, as the sampling error is likely to be smaller than using a systematic
random sample.
NOS will send self-completion survey forms to 6,000 commercial mariners and 4,400 coastal
managers with a link to the research company’s server, so that the survey can be completed
online. This is possible because we have lists of product users’ emails.
The Commercial mariners email will mainly be to subscribers of Professional Mariner, as they
have the most up to date list of chart users. The contact lists below are selected to cover all parts
of the population of chart users. The Coastal Managers email will be to those listed on
[email protected]., which is coordinated and administered by the Delaware Sea Grant
College Program at the University of Delaware.
We show below if each list will be used in its entirety or sampled, and the expected number to
be sampled:
A.
Professional Mariner – a sample of 6,000 subscribers from the total of their mailing list
who are commercial mariners

7
B.
[email protected] – the full list of 4,400 named coastal managers, administered by
the University of Delaware.
2. Describe the procedures for the collection, including: the statistical methodology for
stratification and sample selection; the estimation procedure; the degree of accuracy
needed for the purpose described in the justification; any unusual problems requiring
specialized sampling procedures; and any use of periodic (less frequent than annual) data
collection cycles to reduce burden.
When the lists have been assembled by NOS management, the names of those who will be
contacted will be selected by using a ‘skip interval’, allowing NOS management to go through
the lists and select the names on the basis of the skip interval calculated, using the following
formula:
k=

Size of population
=
Size of sample required

N (Total number in lists provided)
n

In the past use of these lists has provided a high strike rate for contacting chart users. When the
sample is selected, a random choice sequence for label selection (by taking a random number
and counting the intervals of labels) for the contacts will be adopted, counting through the lists
until the sample is full. This has been the procedure used by NOS management in the past that
has ensured a good response and meaningful response.
In surveys of this nature, and in the past NOAA surveys completed, usually, 95% confidence
limits are used for the accurate interpretation of the data - we are confident of using the 95%
level for this survey, as the repeated samplings of the same population of previous surveys
identified the properties of the normal distribution. This provides the confidence limits for 95%
of the samples selected including the parametric mean.
Therefore, analysis of the data will be subjected to a calculation that will be used to calculate
the confidence limits: multiplying the standard error of the mean x the appropriate t-value. This
means that there is a 5% probability or a 1 in 20 chance that the result or finding has occurred
by chance. This is the lowest acceptable level in most market research business to business
projects, such as this survey and will be sufficient for interpreting the data for the guidance
NOS needs.
For example, if we received back 195 questionnaires in the survey for one particular chart user
type (such as Container Ship operators), or 195 commercial mariners say that NOAA service
has improved in the last 12 months, the following calculations could be made:
o
o
o
o

Number Of Observations
Mean
Standard Deviation
Standard Deviation Of Mean

= 195
= 9.261460
= 0.2278881e-01
= 0.1631940e-02

8
Referring to the following table provides the confidence levels that will be used for the survey:
Confidence T
Value
(%)

T X Sd(Mean)
Value

50.000
75.000
90.000
95.000
99.000
99.900
99.990
99.999

0.110279E-02
0.188294E-02
0.269718E-02
0.321862E-02
0.424534E-02
0.545297E-02
0.648365E-02
0.740309E-02

0.676
1.154
1.653
1.972
2.601
3.341
3.973
4.536

Lower
Limit

Upper
Limit
9.26036
9.25958
9.25876
9.25824
9.25721
9.25601
9.25498
9.25406

9.26256
9.26334
9.26416
9.26468
9.26571
9.26691
9.26794
9.26886

The table shows the confidence interval for several different significance levels. The first
column lists the confidence level (which is 1 - expressed as a percent), the second column lists
the t-value, the third column lists the t-value times the standard error, the fourth column lists the
lower confidence limit, and the fifth column lists the upper confidence limit.
Therefore, if we use a 95% confidence interval, the row identified by 95.000 in the first column
then one can see an interval of (9.25824, 9.26468) from the last two columns.
2. Describe the methods used to maximize response rates and to deal with nonresponse.
The accuracy and reliability of the information collected must be shown to be
adequate for the intended uses. For collections based on sampling, a special
justification must be provided if they will not yield "reliable" data that can be
generalized to the universe studied.
Response rates of approximately up to 25% for the surveys are likely to be achieved, based on
previous results and because of the following:
Accuracy of the Email List
NOS has spent time ensuring the accuracy of the email list and the extent to which it is
segmented into the relevant users of NOS charts. This includes those who are regularly
interacting with NOS for all of its services. Use of these lists in the past has also contributed to
the high response of 25% for the surveys.
Re-contacting the Target Audience
Past surveys have taken into account the recent requirement made in all Data Protection and
Codes of Conduct for research, asking the respondents if they would be willing to take part in
future surveys. A high response of 50% has been achieved to this question in the past,
indicating that NOS had contacted the correct target audience and this helps to keep the
response rate high in future surveys, although it is recognised that not all of those responding
affirmatively will take part in the future.

9

The relationship between NOS and its Chart Users
The online self-completion survey has been selected for this survey, because of the relationship
between NOS and Chart users.
Past surveys have shown that the respondents:
•

Are interested in responding, because the survey was sent by NOS

•

Are interested in the subject

•
from

Believe that taking part in the survey will improve the relationship and service received
NOS in the future.

The Cover Email
The questionnaires will be emailed with a cover email signed by a senior member of NOS staff,
which will identify:
•

The purpose of the research

•

The reasons why NOS would like the recipients to respond

•

The ease of completion

•

The short time needed to complete the questionnaire

•

A reminder that no confidentiality can be assured as a result of participating in the
survey, but that respondents can withhold the details of their name and address

•

A NOS and research supplier contact name for more information or questions about the
survey

•

The time scale for return of the questionnaire.

Administration of the NOS Email
In planning this survey we have done our best to ensure that there is a high response to
complete the emailing as follows:
•

Issue an initial email to all the targeted audiences to inform them that a questionnaire
will be emailed shortly and to raise their awareness of its pending arrival.

•

Email the questionnaire with the covering letter 7 days later.

10

•

Email reminding the target respondent to complete survey as soon as possible.

Each survey questionnaire will be checked by the research contractor upon receipt. Any
inaccurate completions will be discarded and only questionnaires which have been completed
accurately and with due consideration will be included in the final analysis.
NOS will implement safeguards throughout the survey production process by asking the
research contractors to ensure that survey data are handled to avoid disclosure. This is done by
separating the survey responses from classification information which would identify who has
provided the response.
Most common apparent reasons for non-response in these surveys are refusals or nonavailability of respondents, likely resulting from the timing of the survey; NOS management is
taking care to schedule the survey during periods not including national holidays.
Mainly because we have not had obviously skewed response profiles in past surveys, we did not
build into the survey, or budget for, any follow-up activities with non-responders.
3. Describe any tests of procedures or methods to be undertaken. Tests are encouraged as
effective means to refine collections, but if ten or more test respondents are involved
OMB must give prior approval.
No tests will be undertaken.
4. Provide the name and telephone number of individuals consulted on the statistical
aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or
other person(s) who will actually collect and/or analyze the information for the agency.
NOS has consulted the appointed contractor for the statistical aspects of the design. The
Contractor is Robin J Birn, President, Strategy, Research and Action, Inc, 29 Lyon, Newport
Coast, CA 92657 Tel: 949-760-3980 Email: [email protected].