Chart Users Survey 2009 supplemental and Part B questions

2009 Nautical Charting Customer Satisfaction Survey
OMB Control No. 0648-0342 – Expires 12/31/2011
Summary
This document provides the details of the 2009 Nautical Charting Customer Satisfaction
Survey, which has been updated from the 2005 survey approved under OMB Control No.
0648-0342.
Explain who will be conducting this survey. What program office will be
conducting the survey? What services does this program provide? Who are the
customers? How are these services provided to the customer?
The survey will be conducted by the National Ocean Service’s Office of Coast Survey
(OCS). OCS produces nautical charts in paper and electronic format, the U.S. Coast Pilot, a
companion publication to nautical charts, and other products supporting marine navigation.
Nautical charts provide navigational coverage of ports and harbors and United States
territorial waters. These nautical products are used by commercial vessels and recreational
boaters transiting United States waters and making calls on seaports. Nautical products are
provided to customers through a government warehouse, a network of approximately 1,800
nautical sales agents, and through the Internet.
The reason for the survey is to act in accordance with Executive Order 12862, the National
Performance Review and good management practices. So NOS is seeking to continue to
determine whether its customers/users are satisfied with the services/products they receive
and whether they have suggestions as to how the services/products may be improved in the
future or made more useful. Therefore the information will be used to get guidance on how
to improve NOS’s products and services.
Explain how this survey was developed. With whom did you consult during the
development of this survey on content? statistics? What suggestions did you get about
improving the survey?
The 2009 survey was developed from prior surveys, one to one interviews and other
comments from users regarding services/products used. The questionnaire for the 2009
survey is based on the questions used in previous surveys, to ensure that there is trend
analysis developed.
Based upon user comments, the length of the survey has been reduced compared to past
surveys, which should increase response, but still provide actionable information. Long
questions asking the respondent to fill out a 5 point scale with statements to rate have been
substituted with shorter questions. There are also short questions about new Chart formats,
which are asking for fast comparison of Chart format options, helping NOS to improve
Charts for mariners in the future.
1

The new design of the paper version of the questionnaire for the Commercial Mariners
survey has enabled the survey questionnaire to be reformatted, resulting in saving two pages
per printed questionnaire, substantially reducing the amount of paper used to print the
questionnaires.
In addition, as the Recreational Mariners questionnaire is being emailed to mariners for
online completion it will not be necessary to print a paper version of this questionnaire,
saving eight pages per printed questionnaire. This version of the questionnaire will only be
able to be filled out by the recipient of the email and they will only be able to fill out the
survey once.
Professional assistance in question development was provided by the survey firm, Strategy,
Research, and Action, who have worked in producing questionnaires which have been
effective amongst Commercial and Recreational chart users in the past.
NOS has ensured that the 2009 survey is designed and administered using the knowledge
gained from previous surveys, which will ensure that the questionnaire achieves the best
balance between maximizing data quality and controlling measurement error, while
minimizing respondent burden and cost.
Explain how the survey will be conducted. How will the customers be sampled (if fewer
than all customers will be surveyed)? What percentage of customers asked to take the
survey will respond? What actions are planned to increase the response rate? (Webbased surveys are not an acceptable method of sampling a broad population. Webbased surveys must be limited to services provided by Web.)
The universe for the sample is defined as ‘users of charts and nautical products relating to
navigating in US charted waters’. The selection of the sample frame for this survey is based
on many years of reviewing Chart users’ lists in relation to how the data is to be used, to give
guidance on how to improve services/products.
NOS believes completing a Census of users is beyond the budget allocation that is available
for the completion of this survey. Therefore NOS has worked with Strategy, Research &
Action to develop a sample to give as representative and unbiased data about Chart users as
possible.
The aim of this survey is to select a representative sample using a stratified sampling
technique, in which the chosen sample is forced to contain potential respondents from each
of the key segments of the population. Such a sample provides statistical efficiency, as the
sampling error is likely to be smaller than using a systematic random sample.
The survey will be conducted by mailing over 3,000 self-completion survey forms to
Commercial mariners and other Commercial users – as was done for the 2005 survey. The
change to the past survey is that in 2009 we will be sending 3,000 emails to Recreational
boaters and other Non-commercial users, together with a link to the research company’s
2

server, so that the survey can be completed on-line (this was done by regular mail last time).
This venue is possible because we have a list which gives named Chart users’ emails. It is
not possible to email Commercial Mariners as the list providers have not collected their email
addresses.
The Commercial mailing will mainly be to licensed professional mariners (listed in the
Masters, Mates, and Pilots Union) and the Recreational boaters mailing will mainly be to
members of the U.S. Power Squadron, a national recreational boating organization, and to
members of Boat US. Both of these organizations have the most up to date list of Chart
users, and their lists have provided an excellent response in the past. Therefore we will be
using the same sampling lists updated by the relevant organizations providing them. The
contact lists below are selected to cover all parts of the population of Chart users.
We show below if each list will be used in its entirety or sampled, and the expected number
to be sampled:
1.

BoatUS – a sample of 3,000 subscribers from the total of their mailing list
who are recreational mariners

2.

American Pilots' association – the total sample list of 1,200 members
(probably most are a subset of Masters and Mates Union)

3.

Local port authorities - The total list of 40 locations and named users of
charts

4.

Masters and Mates Union – a sample of 3,000 members and named chart
users from the total of their membership list

5.

U.S. Power Squadrons – a sample of 2,800 named chart users, participants
in Co-operative Charting Program from the total of their list. (Sample
selection)

6.

U.S. Navy – the full list of 300 ships with named chart users

7.

Tug and Barge Companies – a sample of 5000 tug boats from the full list
of companies

8.

Workboat companies – the full list of 100 organizations with named
operators who are chart users

9.

Lake Carriers Association - the full list of 12 companies with named chart
users

10. US Coast Guard – the full list of 100 named users of charts.
3

(We will not be using the American Waterways Board lists as they overlap with the
lists shown above.)
The total sample of Chart Users is expected to be approximately 6,000, after duplicate
names have been eliminated.
When the lists have been assembled by NOS management, the names of those who will be
contacted will be selected by using a ‘skip interval’, allowing NOS management to go
through the lists and select the names on the basis of the skip interval calculated, using the
following formula:
K = Size of population =
N (Total number in lists provided)
Size of sample required
n
In the past, use of these lists has provided a high strike rate for contacting Chart users. When
the sample is selected, a random choice sequence for label selection (by taking a random
number and counting the intervals of labels) for the contacts will be adopted, counting
through the lists until the sample is full.
The International Research industry recognizes that response rates are not as high as in the
past due to those in business and commerce and Government Agencies having less time than
they used to, because of their priorities and schedules. Therefore all research suppliers have
been asked to advise the commissioning organisation not to expect as high responses as in
the past. The agreed typical industry response rates for well-executed surveys are 10 – 35%,
but most achieve lower responses.
Response rates of approximately 25% for both the Commercial and Recreational surveys are
likely to be achieved, based on previous results and because of the following:
•

Accuracy of the Mailing List
NOS has spent time ensuring the accuracy of the mailing list and the extent to which
it is segmented into the relevant users of NOS charts. This includes those who are
regularly interacting with NOS for all of its services. Use of these lists in the past has
also contributed to the high response of 25% for the surveys.

•

Re-contacting the Target Audience
Past surveys have taken into account the recent requirement made in all Data
Protection and Codes of Conduct for research, asking the respondents if they would
be willing to take part in future surveys. A high response of 50% has been achieved
to this question in the past, indicating that NOS had contacted the correct target
audience and this helps to keep the response rate high in future surveys, although it is
recognised that not all of them will take part in the future.
4

•

The relationship between NOS and its Chart Users
The self-completion survey has been selected for this survey, because of the
relationship between NOS and Chart users.
Past surveys have shown that the respondents:
o Are interested in responding, because the survey was sent by NOS
o Are interested in the subject
o Believe that taking part in the survey will improve the relationship and service
received from NOS in the future.

•

The Cover Letter for the NOS Mailing and emailing
The questionnaires will be mailed and emailed with a cover letter/email signed by a
senior member of NOS staff, which will identify:
o The purpose of the research
o The reasons why NOS would like the recipients to respond
o The ease of completion
o The short time needed to complete the questionnaire
o A reminder that no confidentiality can be assured as a result of participating in
the survey, but that respondents can withhold the details of their name and
address
o A NOS and research supplier contact name for more information or questions
about the survey
o The time scale for return of the questionnaire, pointing out that a pre-paid
envelope is also provided or survey link is provided for on-line completion.

•

Administration of the NOS Mailing/Emailing
In planning this survey, we have done our best to ensure that there is a high response
by taking the following steps:
o Issue an initial letter/email to all the targeted audiences to inform them that a
voluntary questionnaire will be mailed/emailed shortly and to raise their
awareness of its pending arrival.
o Mail/email the questionnaire with the covering letter 7 days later. NOS will
inform respondents that the responses to the collection of information are
5

voluntary. The information will also state that the questionnaire will take no
longer than 10 minutes to complete. The OMB Control Number and a
Paperwork Reduction Act Statement will be shown at the end of both the
paper and on-line questionnaire.
o Mail/email a postcard 7 days after the questionnaire has been mailed/emailed
reminding the target respondent to return the questionnaire as soon as
possible. On one side of this postcard will be printed one of NOAA’s
attractive posters, to catch the attention of the recipient and increase the
chances of his/her reading the card.
Each survey questionnaire will be checked by the research contractor upon receipt. Any
inaccurate completions will be discarded and only questionnaires which have been completed
accurately and with due consideration will be included in the final analysis.
NOS will implement safeguards throughout the survey production process by asking the
research contractors to ensure that survey data are handled to avoid disclosure. This is done
by separating the survey responses from classification information which would identify who
has provided the response.
Non-response bias is known to lead to results that misrepresent the targeted population. In
past surveys, there has been a relatively balanced response over all user types
(professional/commercial and recreational, as well as vessel types); that is, approximately the
same percentage of response over all categories.
Most common apparent reasons for non-response in these surveys are refusals or nonavailability of respondents, likely resulting from the timing of the survey; NOS management
is taking care to schedule the survey during periods not including national holidays. In
addition, previous surveys have shown that some non-response is due to mariners being away
at sea, and as a result, larger samples will be drawn from some of the lists, to partially offset
this factor. The fact that responses are sometimes received several months after the survey
period supports the conclusion that non-availability is a factor, while also demonstrating that
these late respondents see themselves as stakeholders.
Mainly because we have not had obviously skewed response profiles in past surveys, we did
not build into the survey, or budget for, any follow-up activities with non-responders.
Describe how the results of this survey will be analyzed and used. If the customer
population is sampled, what statistical techniques will be used to generalize the results
to the entire customer population? Is this survey intended to measure a GPRA
performance measure? (If so, please include an excerpt from the appropriate
document.)

6

Analysis and Report Planning
A multivariate analysis will be conducted on the responses and Chart user types and other
chosen classification information, such as affiliations. Means and standard deviations and
standard errors for user representation will also be calculated. The survey is not intended to
measure a GPRA performance measure. These analyses will be comparable with previous
surveys so that similar statistical conclusions can be derived from the 2009 survey data as is
acceptable statistical practice.
Review of Information Products
NOS will be responsible for the quality of information that they disseminate and will institute
appropriate content/subject matter, statistical and methodological review procedures to
comply with OMB and agency Information Quality Guidelines.
Releasing Information
NOS will produce a research summary which can be used to release information intended for
the general public in line with NOS dissemination policies and procedures. An Appendix will
describe the procedures that have been used to evaluate the quality of the data, to allow users
to interpret results of analyses and to help designers of recurring surveys to focus on
improvement efforts.
Data Protection and Disclosure Avoidance for Dissemination
NOS will implement safeguards throughout the survey production process by asking the
research contractors to ensure that survey data are handled to avoid disclosure. This is done
by separating the survey responses from classification information which would identify who
has provided the response.
NOS will ensure that the information that is published or released on request will comply
with applicable Federal legislation and regulations.

Survey Documentation
NOS will produce survey documentation that includes those materials needed to analyze data
from the survey, as well as the information necessary to replicate and evaluate the survey’s
results compared with previous surveys.
Documentation and Release of Public-Use Microdata
If NOS releases microdata to the public it will ensure that it will include documentation
clearly describing how the information is constructed and if required will also provide the
metadata necessary for users to access and manipulate the data.
7

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
Describe how the results of this survey will be analyzed and used. If the customer
population is sampled, what statistical techniques will be used to generalize the results
to the entire customer population? Is this survey intended to measure a GPRA
performance measure? (If so, please include an excerpt from the appropriate
document.)
The universe for the sample is defined as ‘users of charts and nautical products relating to
navigating in US charted waters’. The aim of this survey is to select a representative sample
using a stratified sampling technique, in which the chosen sample is forced to contain
potential respondents from each of the key segments of the population. Such a sample
provides statistical efficiency, as the sampling error is likely to be smaller than using a
systematic random sample.
The survey will be conducted by mailing over 3,000 self-completion survey forms to
commercial mariners and other commercial users – as was done for the 2005 survey. The
change to the past survey is that in 2009 we will be sending 3,000 emails to recreational
boaters and other non-commercial users, together with a link to the research company’s
server, so that the survey can be completed on-line (this was done by regular mail last time).
It is not possible to email commercial mariners as the list providers have not collected their
email addresses.
The commercial mailing will mainly be to licensed professional mariners (listed in the
Masters, Mates, and Pilots Union) and the Recreational boaters mailing will mainly be to
members of the U.S. Power Squadron, a national recreational boating organization, and to
members of Boat US. Both of these organizations have the most up to date list of chart users,
and their lists have provided an excellent response in the past. Therefore we will be using the
same sampling lists updated by the relevant organizations providing them. The contact lists
below are selected to cover all parts of the population of chart users.
We show below if each list will be used in its entirety or sampled, and the expected number
to be sampled:
1. BoatUS – a sample of 3,000 subscribers from the total of their mailing list who
are recreational mariners
2. American Pilots' association – the total sample list of 1,200 members (probably
most are a subset of Masters and Mates Union)
3. Local port authorities - The total list of 40 locations and named users of charts.
4. Masters and Mates Union – a sample of 3,000 members and named chart users
from the total of their membership list
8

5. U.S. Power Squadrons – a sample of 2,800 named chart users, participants in Cooperative Charting Program from the total of their list. (Sample selection)
6. U.S. Navy – the full list of 300 ships with named chart users.
7. Tug and Barge Companies – a sample of 5000 tug boats from the full list of
companies
8. Workboat companies – the full list of 100 organizations with named operators
who are chart users.
9. Lake Carriers Association - the full list of 12 companies with named chart users.
10.

US Coast Guard – the full list of 100 named users of charts.

(We will not be using the American Waterways Board lists as they overlap with the
lists shown above.)
The total sample of Chart Users is expected to be approximately 6,000, after duplicate
names have been eliminated.
Describe the procedures for the collection, including: the statistical methodology for
stratification and sample selection; the estimation procedure; the degree of accuracy
needed for the purpose described in the justification; any unusual problems requiring
specialized sampling procedures; and any use of periodic (less frequent than annual)
data collection cycles to reduce burden.
When the lists have been assembled by NOS management, the names of those who will be
contacted will be selected by using a ‘skip interval’, allowing NOS management to go
through the lists and select the names on the basis of the skip interval calculated, using the
following formula:
k=

Size of population
=
Size of sample required

N (Total number in lists provided)
n

In the past use of these lists has provided a high strike rate for contacting chart users. When
the sample is selected, a random choice sequence for label selection (by taking a random
number and counting the intervals of labels) for the contacts will be adopted, counting
through the lists until the sample is full. This has been the procedure used by NOS
management in the past that has ensured a good response and meaningful response.
In surveys of this nature, and in the past NOAA surveys completed, usually, 95% confidence
limits are used for the accurate interpretation of the data - we are confident of using the 95%
level for this survey, as the repeated samplings of the same population of previous surveys
9

identified the properties of the normal distribution. This provides the confidence limits for
95% of the samples selected including the parametric mean.
Therefore, analysis of the data will be subjected to a calculation that will be used to calculate
the confidence limits: multiplying the standard error of the mean x the appropriate t-value.
This means that there is a 5% probability or a 1 in 20 chance that the result or finding has
occurred by chance. This is the lowest acceptable level in most market research business to
business projects, such as this survey and will be sufficient for interpreting the data for the
guidance NOS needs.
For example, if we received back 195 questionnaires in the survey for one particular chart
user type (such as Tug Boat operators), or 195 mariners say that NOAA service has improved
in the last 12 months, the following calculations could be made:
o
o
o
o

Number Of Observations
Mean
Standard Deviation
Standard Deviation Of Mean

= 195
= 9.261460
= 0.2278881e-01
= 0.1631940e-02

Referring to the following table provides the confidence levels that will be used for the
survey:
Confidence T
Value
(%)
50.000
75.000
90.000
95.000
99.000
99.900
99.990
99.999

0.676
1.154
1.653
1.972
2.601
3.341
3.973
4.536

T X Sd(Mean)
Value
0.110279E-02
0.188294E-02
0.269718E-02
0.321862E-02
0.424534E-02
0.545297E-02
0.648365E-02
0.740309E-02

Lower
Limit

9.26036
9.25958
9.25876
9.25824
9.25721
9.25601
9.25498
9.25406

Upper
Limit

9.26256
9.26334
9.26416
9.26468
9.26571
9.26691
9.26794
9.26886

The table shows the confidence interval for several different significance levels. The first
column lists the confidence level (which is 1 - expressed as a percent), the second column
lists the t-value, the third column lists the t-value times the standard error, the fourth column
lists the lower confidence limit, and the fifth column lists the upper confidence limit.
Therefore, if we use a 95% confidence interval, the row identified by 95.000 in the first
column then you can see an interval of (9.25824, 9.26468) from the last two columns.

10

Describe the methods used to maximize response rates and to deal with nonresponse.
The accuracy and reliability of the information collected must be shown to be adequate
for the intended uses. For collections based on sampling, a special justification must be
provided if they will not yield "reliable" data that can be generalized to the universe
studied.
Response rates of approximately 25% for both the Commercial and Recreational surveys are
likely to be achieved, based on previous results and because of the following:
NOS has used its experience from past surveys to make the questions and the questionnaire
as short and as attractive as possible with reply paid envelopes provided or on-line
connection for fast completion.
Accuracy of the Mailing List
NOS has spent time ensuring the accuracy of the mailing list and the extent to which it is
segmented into the relevant users of NOS charts. This includes those who are regularly
interacting with NOS for all of its services. Use of these lists in the past has also contributed
to the high response of 25% for the surveys.
Re-contacting the Target Audience
Past surveys have taken into account the recent requirement made in all Data Protection and
Codes of Conduct for research, asking the respondents if they would be willing to take part in
future surveys. A high response of 50% has been achieved to this question in the past,
indicating that NOS had contacted the correct target audience and this helps to keep the
response rate high in future surveys, although it is recognised that not all of them will take
part in the future.
The relationship between NOS and its Chart Users
The self-completion survey has been selected for this survey, because of the relationship
between NOS and Chart users.
Past surveys have shown that the respondents:
•

Are interested in responding, because the survey was sent by NOS

•

Are interested in the subject

•

Believe that taking part in the survey will improve the relationship and service
received from NOS in the future.

11

In the past, NOS has received a small amount of questionnaires back long after the survey
had been completed, when the Mariners have returned to base; thus showing that the survey
was relevant and one that they wanted to take part in.
The Cover Letter for the NOS Mailing and emailing
The questionnaires will be mailed and emailed with a cover letter/email signed by a senior
member of NOS staff, which will identify:
•

The purpose of the research

•

The reasons why NOS would like the recipients to respond

•

The ease of completion

•

The short time needed to complete the questionnaire

•

A reminder that no confidentiality can be assured as a result of participating in the
survey, but that respondents can withhold the details of their name and address

•

A NOS and research supplier contact name for more information or questions about
the survey

•

The time scale for return of the questionnaire, pointing out that a pre-paid envelope is
also provided or survey link is provided for on-line completion.

Administration of the NOS Mailing/Emailing
In planning this survey we have done our best to ensure that there is a high response to
complete the mailing as follows:
•

Issue an initial letter/email to all the targeted audiences to inform them that a
questionnaire will be mailed/emailed shortly and to raise their awareness of its
pending arrival.

•

Mail/email the questionnaire with the covering letter 7 days later.

•

Mail/email a postcard 7 days after the questionnaire has been mailed/emailed
reminding the target respondent to return the questionnaire as soon as possible. On
one side of this postcard will be printed one of NOAA’s attractive posters, to catch
the attention of the recipient and increase the chances of his/her reading the card.

12

Each survey questionnaire will be checked by the research contractor upon receipt. Any
inaccurate completions will be discarded and only questionnaires which have been completed
accurately and with due consideration will be included in the final analysis.
NOS will implement safeguards throughout the survey production process by asking the
research contractors to ensure that survey data are handled to avoid disclosure. This is done
by separating the survey responses from classification information which would identify who
has provided the response.
Non-response bias is known to lead to results that misrepresent the targeted population. In
past Chart Users surveys, there has been a relatively balanced response over all user types
(professional/commercial and recreational, as well as vessel types); that is, approximately the
same percentage of response over all categories.
Most common apparent reasons for non-response in these surveys are refusals or nonavailability of respondents, likely resulting from the timing of the survey; NOS management
is taking care to schedule the survey during periods not including national holidays. In
addition, previous surveys have shown that some non-response is due to mariners being away
at sea, and as a result, larger samples will be drawn from some of the lists, to partially offset
this factor. The fact that responses are sometimes received several months after the survey
period supports the conclusion that non-availability is a factor, while also demonstrating that
these late respondents see themselves as stakeholders.
Mainly because we have not had obviously skewed response profiles in past surveys, we did
not build into the survey, or budget for, any follow-up activities with non-responders.
Describe any tests of procedures or methods to be undertaken. Tests are encouraged as
effective means to refine collections, but if ten or more test respondents are involved
OMB must give prior approval.
No additional tests will be undertaken.
Provide the name and telephone number of individuals consulted on the statistical
aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or
other person(s) who will actually collect and/or analyze the information for the agency.
NOS has consulted the appointed contractor for the statistical aspects of the design. The
Contractor is Robin J Birn, President, Strategy, Research and Action, Inc, 29 Lyon, Newport
Coast, CA 92657 Tel: 949-760-3980 Email: [email protected].

13