Chart Users Supplementary and Part B Questions

1

2010 Nautical Charting Customer Satisfaction Survey
OMB Control No. 0648-0342 – Expires 12/31/2011
Summary
This document provides the details of the 2010 Nautical Charting Customer
Satisfaction Survey, which has been updated from the 2009 survey approved under
OMB Control No. 0648-0342.
Explain who will be conducting this survey. What program office will be
conducting the survey? What services does this program provide? Who are
the customers? How are these services provided to the customer?
The survey will be conducted by the National Ocean Service’s (NOS) Office of Coast
Survey (OCS). OCS produces nautical charts in paper and electronic format, the
U.S. Coast Pilot, a companion publication to nautical charts, and other products
supporting marine navigation. Nautical charts provide navigational coverage of ports
and harbors and United States territorial waters. These nautical products are used
by commercial vessels and recreational boaters transiting United States waters and
making calls on seaports. Nautical products are provided to customers through a
government warehouse, a network of approximately 1,800 nautical sales agents,
and through the Internet.
The reason for the survey is to act in accordance with Executive Order 12862, the
National Performance Review and good management practices. NOS is seeking to
continue to determine whether its customers/users are satisfied with the
services/products they receive and whether they have suggestions as to how the
information collected will be used to get guidance on how to improve NOS’s
products and services.
The current NOAA strategy focuses on the non-traditional users of NOAA’s products
and services. Therefore, the 2010 survey will be focused on the largest group of
these users, who are recreational mariners, to get to know their needs better and
obtain more feedback about the needs of small craft sailers.
Explain how this survey was developed. With whom did you consult during
the development of this survey on content? statistics? What suggestions did
you get about improving the survey?
The 2010 survey was developed from prior surveys and the questionnaire is based
on the questions used in previous surveys, to ensure that there is trend analysis
developed.

2010 Nautical Charting Customer Satisfaction Survey 2010
Order Number: DG133CO9SE4763
Requisition Number: NCNJ50000-9-17057

2
Based upon user comments, the length of the survey has been reduced compared
to past surveys, which should increase the response rate, but still provide actionable
information.
In the 2010 survey, the recreational mariners will simply answer questions; there will
not be any visual representation of charts to view and provide feedback as in 2009.
The survey will be administered online and there will not be any paper version of the
survey printed in 2010. This online version of the questionnaire will be able to be
filled out only by the recipient of the email and he/she will only be able to fill out the
survey once.
Professional assistance in question development has been provided by the survey
firm, Strategy, Research, and Action, who has worked in producing questionnaires
which have provided good data on Recreational chart users in the past. NOS has
ensured that the 2010 survey is designed and administered using the knowledge
gained from previous surveys, which will ensure that the questionnaire achieves the
best balance between maximizing data quality and controlling measurement error.
This has helped to design the survey to minimize respondent burden and cost.
Explain how the survey will be conducted. How will the customers be
sampled (if fewer than all customers will be surveyed)? What percentage of
customers asked to take the survey will respond? What actions are planned to
increase the response rate? (Web-based surveys are not an acceptable
method of sampling a broad population. Web-based surveys must be limited
to services provided by Web.)
The universe for the sample is defined as ‘users of charts and nautical products
relating to navigating in US charted waters’. The selection of the sample frame for
this survey is based on many years of reviewing Chart users’ lists in relation to how
the data is to be used, to give guidance on how to improve services/products.
NOS believes completing a Census of users is beyond the budget allocation that is
available for the completion of this survey. Therefore NOS has worked with
Strategy, Research & Action to develop a sample to give as representative and
unbiased data about Chart users as possible.
The aim of this survey is to select a representative sample using a stratified
sampling technique, in which the chosen sample is forced to contain potential
respondents from each of the key segments of the population. Such a sample
provides statistical efficiency, as the sampling error is likely to be smaller than using
a systematic random sample.
The survey will be conducted by mailing over 8,800 self-completion survey forms to
recreational boaters who sail small craft. As in 2009, the emails will be sent to
recreational boaters with a link to the research company’s server, so that the survey
2010 Nautical Charting Customer Satisfaction Survey 2010
Order Number: DG133CO9SE4763
Requisition Number: NCNJ50000-9-17057

3
can be completed online. This is possible because we have a list which gives named
Chart users’ emails.
The Recreational boaters email will mainly be to members of the U.S. Power
Squadron, a national recreational boating organization, and to members of Boat US.
Both of these organizations have the most up to date list of Chart users, and their
lists have provided an excellent response in the past. Therefore we will be using the
same sampling lists as in 2009, updated by the relevant organizations providing
them. The contact lists below are selected to cover all parts of the population of
Chart users.
We show below if each list will be used in its entirety or sampled, and the expected
number to be sampled:
1.

BoatUS – a sample of 6,000 subscribers from the total of their
mailing list who are recreational mariners

2.

U.S. Power Squadrons – a sample of 2,800 named chart users,
participants in Co-operative Charting Program from the total of their
list.

When the lists have been assembled by NOS management, the names of those
who will be contacted will be selected by using a ‘skip interval’, allowing NOS
management to go through the lists and select the names on the basis of the skip
interval calculated, using the following formula:
K = Size of population =
N (Total number in lists provided)
Size of sample required
n
In the past, use of these lists has provided a high strike rate for contacting Chart
users. When the sample is selected, a random choice sequence for label selection
(by taking a random number and counting the intervals of labels) for the contacts will
be adopted, counting through the lists until the sample is full.
The International Research industry recognizes that response rates are not as high
as in the past due to those in business and commerce and Government Agencies
having less time than they used to, because of their priorities and schedules.
Therefore all research suppliers have been asked to advise the commissioning
organisation not to expect as high responses as in the past. The agreed typical
industry response rates for well-executed surveys are 10 – 35%.
In 2009 a response rate of 15% was achieved. A response rate of approximately
25% for the recreational survey is likely to be achieved, based on previous results
and because of the following:

2010 Nautical Charting Customer Satisfaction Survey 2010
Order Number: DG133CO9SE4763
Requisition Number: NCNJ50000-9-17057

4

•

Accuracy of the Email List
NOS has spent time ensuring the accuracy of the email list and the extent to
which it is segmented into the relevant users of NOS charts. This includes
those who are regularly interacting with NOS for all of its services.
Use of these lists in the past has also contributed to the high response of 25%
for the surveys.

•

Re-contacting the Target Audience
Past surveys have taken into account the recent requirement made in all Data
Protection and Codes of Conduct for research, asking the respondents if they
would be willing to take part in future surveys. A high response of 50% has
been achieved to this question in the past, indicating that NOS had contacted
the correct target audience and this helps to keep the response rate high in
future surveys, although it is recognised that not all of them will take part in
the future.

•

The relationship between NOS and its Chart Users
The online self-completion survey has been selected for this survey, because
of the relationship between NOS and Chart users.
Past surveys have shown that the respondents:
o Are interested in responding, because the survey was sent by NOS
o Are interested in the subject
o Believe that taking part in the survey will improve the relationship and
service received from NOS in the future.

•

The Cover email
The questionnaires will be emailed with a cover email signed by a senior
member of NOS staff, which will identify:
o The purpose of the research
o The reasons why NOS would like the recipients to respond
o The ease of completion
o The short time needed to complete the questionnaire
o A reminder that no confidentiality can be assured as a result of
participating in the survey, but that respondents can withhold the
details of their name and address

2010 Nautical Charting Customer Satisfaction Survey 2010
Order Number: DG133CO9SE4763
Requisition Number: NCNJ50000-9-17057

5

o A NOS and research supplier contact name for more information or
questions about the survey
o The time scale for return of the questionnaire.
•

Administration of the NOS Email
In planning this survey, we have done our best to ensure that there is a high
response by taking the following steps:
o Issue an initial email to all the targeted audiences to inform them that a
voluntary questionnaire will be emailed shortly and to raise their
awareness of its pending arrival.
o Email the questionnaire 7 days later. NOS will inform respondents that
the responses to the collection of information are voluntary. The
information will also state that the questionnaire will take no longer
than 10 minutes to complete. The OMB Control Number and a
Paperwork Reduction Act Statement will be shown at the end of both
the paper and on-line questionnaire.
o Email a reminder 7 days after the questionnaire has been emailed
reminding the target respondent to return the questionnaire as soon as
possible.

Each survey questionnaire will be checked by the research contractor upon receipt.
Any inaccurate completions will be discarded and only questionnaires which have
been completed accurately and with due consideration will be included in the final
analysis.
NOS will implement safeguards throughout the survey production process by asking
the research contractors to ensure that survey data are handled to avoid disclosure.
This is done by separating the survey responses from classification information,
which would identify who has provided the response.
Most common apparent reasons for non-response in these surveys are refusals or
non-availability of respondents, likely resulting from the timing of the survey; NOS
management is taking care to schedule the survey during periods not including
national holidays.
Mainly because we have not had obviously skewed response profiles in past
surveys, we did not build into the survey, or budget for, any follow-up activities with
non-responders.

2010 Nautical Charting Customer Satisfaction Survey 2010
Order Number: DG133CO9SE4763
Requisition Number: NCNJ50000-9-17057

6

Describe how the results of this survey will be analyzed and used. If the
customer population is sampled, what statistical techniques will be used to
generalize the results to the entire customer population? Is this survey
intended to measure a GPRA performance measure? (If so, please include an
excerpt from the appropriate document.)
Analysis and Report Planning
A multivariate analysis will be conducted on the responses and Chart user types and
other chosen classification information, such as affiliations. Means and standard
deviations and standard errors for user representation will also be calculated. The
survey is not intended to measure a GPRA performance measure. These analyses
will be comparable with previous surveys so that similar statistical conclusions can
be derived from the 2009 survey data as is acceptable statistical practice.
Review of Information Products
NOS will be responsible for the quality of information that they disseminate and will
institute appropriate content/subject matter, statistical and methodological review
procedures to comply with OMB and agency Information Quality Guidelines.
Releasing Information
NOS will produce a research summary which can be used to release information
intended for the general public in line with NOS dissemination policies and
procedures. An Appendix will describe the procedures that have been used to
evaluate the quality of the data, to allow users to interpret results of analyses and to
help designers of recurring surveys to focus on improvement efforts.
Data Protection and Disclosure Avoidance for Dissemination
NOS will implement safeguards throughout the survey production process by asking
the research contractors to ensure that survey data are handled to avoid disclosure.
This is done by separating the survey responses from classification information
which would identify who has provided the response.
NOS will ensure that the information that is published or released on request will
comply with applicable Federal legislation and regulations.
Survey Documentation
NOS will produce survey documentation that includes those materials needed to
analyze data from the survey, as well as the information necessary to replicate and
evaluate the survey’s results compared with previous surveys.
2010 Nautical Charting Customer Satisfaction Survey 2010
Order Number: DG133CO9SE4763
Requisition Number: NCNJ50000-9-17057

7

Documentation and Release of Public-Use Microdata
If NOS releases microdata to the public it will ensure that it will include
documentation clearly describing how the information is constructed and if required
will also provide the metadata necessary for users to access and manipulate the
data.
B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
Describe (including a numerical estimate) the potential respondent universe and any
sampling or other respondent selection method to be used. Data on the number of
entities (e.g. establishments, State and local governmental units, households, or persons)
in the universe and the corresponding sample are to be provided in tabular form. The
tabulation must also include expected response rates for the collection as a whole. If the
collection has been conducted before, provide the actual response rate achieved.
The universe for the sample is defined as ‘users of charts and nautical products
relating to navigating in US charted waters’. The aim of this survey is to select a
representative sample using a stratified sampling technique, in which the chosen
sample is forced to contain potential respondents from each of the key segments of
the population. Such a sample provides statistical efficiency, as the sampling error
is likely to be smaller than using a systematic random sample.
The survey will be conducted by mailing over 8,800 self-completion survey forms to
recreational boaters, together with a link to the research company’s server, so that
the survey can be completed on-line. The Recreational boaters emailing will mainly
be to members of the U.S. Power Squadron, a national recreational boating
organization, and to members of Boat US. Both of these organizations have the
most up to date list of chart users, and their lists have provided an excellent
response in the past. Therefore we will be using the same sampling lists updated by
the relevant organizations providing them. The contact lists below are selected to
cover all parts of the population of chart users.
We show below if each list will be used in its entirety or sampled, and the expected
number to be sampled:
1. BoatUS – a sample of 6,000 subscribers from the total of their mailing list
who are recreational mariners
2. U.S. Power Squadrons – a sample of 2,800 named chart users,
participants in Co-operative Charting Program from the total of their list.

2010 Nautical Charting Customer Satisfaction Survey 2010
Order Number: DG133CO9SE4763
Requisition Number: NCNJ50000-9-17057

8

In 2009 the Recreational survey sample achieved a response rate of 15% - as we
are increasing the email in 2010 we hope to improve this response.
Describe the procedures for the collection, including: the statistical
methodology for stratification and sample selection; the estimation procedure;
the degree of accuracy needed for the purpose described in the justification;
any unusual problems requiring specialized sampling procedures; and any
use of periodic (less frequent than annual) data collection cycles to reduce
burden.
When the lists have been assembled by NOS management, the names of those who
will be contacted will be selected by using a ‘skip interval’, allowing NOS
management to go through the lists and select the names on the basis of the skip
interval calculated, using the following formula:
k=

Size of population =
Size of sample required

N (Total number in lists provided)
n

In the past use of these lists has provided a high strike rate for contacting chart
users. When the sample is selected, a random choice sequence for label selection
(by taking a random number and counting the intervals of labels) for the contacts will
be adopted, counting through the lists until the sample is full. This has been the
procedure used by NOS management in the past that has ensured a good response
and meaningful response.
In surveys of this nature, and in the past NOAA surveys completed, usually, 95%
confidence limits are used for the accurate interpretation of the data - we are
confident of using the 95% level for this survey, as the repeated samplings of the
same population of previous surveys identified the properties of the normal
distribution. This provides the confidence limits for 95% of the samples selected
including the parametric mean.
Therefore, analysis of the data will be subjected to a calculation that will be used to
calculate the confidence limits: multiplying the standard error of the mean x the
appropriate t-value. This means that there is a 5% probability or a 1 in 20 chance
that the result or finding has occurred by chance. This is the lowest acceptable level
in most market research business to business projects, such as this survey and will
be sufficient for interpreting the data for the guidance NOS needs.
For example, if we received back 195 questionnaires in the survey for one particular
chart user type (such as Sail Boat operators), or 195 recreational mariners say that
NOAA service has improved in the last 12 months, the following calculations could
be made:
o Number Of Observations
2010 Nautical Charting Customer Satisfaction Survey 2010
Order Number: DG133CO9SE4763
Requisition Number: NCNJ50000-9-17057

= 195

9
o Mean
o Standard Deviation
o Standard Deviation Of Mean

= 9.261460
= 0.2278881e-01
= 0.1631940e-02

Referring to the following table provides the confidence levels that will be used for
the survey:
Confidence T
Value
(%)

T X Sd(Mean)
Value

Lower
Limit

Upper
Limit

50.000
75.000
90.000
95.000
99.000
99.900
99.990
99.999

0.110279E-02
0.188294E-02
0.269718E-02
0.321862E-02
0.424534E-02
0.545297E-02
0.648365E-02
0.740309E-02

9.26036
9.25958
9.25876
9.25824
9.25721
9.25601
9.25498
9.25406

9.26256
9.26334
9.26416
9.26468
9.26571
9.26691
9.26794
9.26886

0.676
1.154
1.653
1.972
2.601
3.341
3.973
4.536

The table shows the confidence interval for several different significance levels. The
first column lists the confidence level (which is 1 - expressed as a percent), the
second column lists the t-value, the third column lists the t-value times the standard
error, the fourth column lists the lower confidence limit, and the fifth column lists the
upper confidence limit.
Therefore, if we use a 95% confidence interval, the row identified by 95.000 in the
first column then you can see an interval of (9.25824, 9.26468) from the last two
columns.
Describe the methods used to maximize response rates and to deal with
nonresponse. The accuracy and reliability of the information collected must be
shown to be adequate for the intended uses. For collections based on
sampling, a special justification must be provided if they will not yield
"reliable" data that can be generalized to the universe studied.
In 2009 we achieved a response rate of 15%. Response rates of approximately
25% for the Recreational survey are likely to be achieved, based on previous results
and because of the following:
Accuracy of the Email List
NOS has spent time ensuring the accuracy of the email list and the extent to which it
is segmented into the relevant users of NOS charts. This includes those who are
regularly interacting with NOS for all of its services. Use of these lists in the past
has also contributed to the high response of 25% for the surveys.
2010 Nautical Charting Customer Satisfaction Survey 2010
Order Number: DG133CO9SE4763
Requisition Number: NCNJ50000-9-17057

10

Re-contacting the Target Audience
Past surveys have taken into account the recent requirement made in all Data
Protection and Codes of Conduct for research, asking the respondents if they would
be willing to take part in future surveys.
A high response of 50% has been achieved to this question in the past, indicating
that NOS had contacted the correct target audience and this helps to keep the
response rate high in future surveys, although it is recognised that not all of them will
take part in the future.
The relationship between NOS and its Chart Users
The online self-completion survey has been selected for this survey, because of the
relationship between NOS and Chart users.
Past surveys have shown that the respondents:
•

Are interested in responding, because the survey was sent by NOS

•

Are interested in the subject

•

Believe that taking part in the survey will improve the relationship and service
received from NOS in the future.

The Cover Email
The questionnaires will be emailed with a cover email signed by a senior member of
NOS staff, which will identify:
•

The purpose of the research

•

The reasons why NOS would like the recipients to respond

•

The ease of completion

•

The short time needed to complete the questionnaire

•

A reminder that no confidentiality can be assured as a result of participating in
the survey, but that respondents can withhold the details of their name and
address

•

A NOS and research supplier contact name for more information or questions
about the survey

2010 Nautical Charting Customer Satisfaction Survey 2010
Order Number: DG133CO9SE4763
Requisition Number: NCNJ50000-9-17057

11

•

The time scale for return of the questionnaire.

Administration of the NOS Email
In planning this survey we have done our best to ensure that there is a high
response to complete the emailing as follows:
•

Issue an initial email to all the targeted audiences to inform them that a
questionnaire will be emailed shortly and to raise their awareness of its
pending arrival.

•

Email the questionnaire with the covering letter 7 days later.

•

Email reminding the target respondent to complete survey as soon as
possible.

Each survey questionnaire will be checked by the research contractor upon receipt.
Any inaccurate completions will be discarded and only questionnaires which have
been completed accurately and with due consideration will be included in the final
analysis.
NOS will implement safeguards throughout the survey production process by asking
the research contractors to ensure that survey data are handled to avoid disclosure.
This is done by separating the survey responses from classification information
which would identify who has provided the response.
Most common apparent reasons for non-response in these surveys are refusals or
non-availability of respondents, likely resulting from the timing of the survey; NOS
management is taking care to schedule the survey during periods not including
national holidays.
Mainly because we have not had obviously skewed response profiles in past
surveys, we did not build into the survey, or budget for, any follow-up activities with
non-responders as there were no problems experienced in 2009
Describe any tests of procedures or methods to be undertaken. Tests are
encouraged as effective means to refine collections, but if ten or more test
respondents are involved OMB must give prior approval.
No additional tests will be undertaken.

2010 Nautical Charting Customer Satisfaction Survey 2010
Order Number: DG133CO9SE4763
Requisition Number: NCNJ50000-9-17057

12

Provide the name and telephone number of individuals consulted on the
statistical aspects of the design, and the name of the agency unit,
contractor(s), grantee(s), or other person(s) who will actually collect and/or
analyze the information for the agency.
NOS has consulted the appointed contractor for the statistical aspects of the design.
The Contractor is Robin J Birn, President, Strategy, Research and Action, Inc, 29
Lyon, Newport Coast, CA 92657 Tel: 949-760-3980 Email:
[email protected].

Robin J Birn – 9th February 2010

2010 Nautical Charting Customer Satisfaction Survey 2010
Order Number: DG133CO9SE4763
Requisition Number: NCNJ50000-9-17057