Supplementary and Part B Questions

10

OMB Control No. 0648-0342

Expiration Date: 4/30/2015

Summary

This document provides the details of the Nautical Charting Customer Satisfaction Survey, 2014 version, with one new question, and minor changes for clarity for the previous questions from the 2012 survey approved under OMB Control No. 0648-0342.

1. Explain who will be conducting this survey. What program office will be conducting the survey? What services does this program provide? Who are the customers? How are these services provided to the customer?

The survey will be conducted by the National Ocean Service’s Office of Coast Survey (OCS). OCS produces nautical charts in paper and electronic format, the U.S. Coast Pilot, a companion publication to nautical charts, and other products supporting marine navigation. Nautical charts provide navigational coverage of ports and harbors and United States territorial waters. These nautical products are used by commercial vessels and recreational boaters transiting United States waters and making calls on seaports.

This survey is in accordance with Executive Order 12862, the National Performance Review and good management practices. NOS is seeking to continue to determine whether its customers/users are satisfied with the services/products they receive and whether they have suggestions as to how the services/products may be improved in the future or made more useful. Therefore the information will be used to obtain guidance on how to improve NOS’s products and services.

The current NOAA strategy focuses on traditional and non-traditional users of NOAA’s products and services. The 2014 survey will be focused on both groups of traditional users, who are commercial and recreational mariners, to get to know their needs better and obtain more feedback about commercial and recreational craft sailing. A question has been added regarding a new United States Coast Guard (USCG) requirement: In 2018 they will begin to require all vessels of a certain size to carry a certain chart display system, Electronic Chart Display and Information System (ECDIS).  We are asking a question about how prepared mariners are for this change and if there is anything we can do to help them.  Included is a table to provide the person taking the survey with additional information about upcoming ECDIS requirements. 

2. Explain how this survey was developed. With whom did you consult during the development of this survey on content? Statistics? What suggestions did you get about improving the survey?

The 2014 commercial and recreational mariner’s survey questions are based on key trend questions used in previous surveys, to ensure that there is trend analysis developed and so new topics which will aid NOS to plan new products and services. Based upon user comments, the length of the questionnaire for the survey has again been reduced compared to past surveys, which should increase response, but still provide actionable information.

In the 2014 survey the commercial and recreational mariners will answer only direct questions, and the survey will be administered online. This online version of the questionnaire will be able to be filled out only by the recipient of the email and he/she will be able to fill out the survey only once.

Professional assistance in question development has been provided by the survey firm, Strategy, Research, and Action, who have contributed to producing questionnaires which have been effective amongst chart users in the past. The 2014 survey is designed and will be administered using the knowledge gained from previous surveys, which will ensure that the questionnaire achieves the best balance between maximizing data quality and controlling measurement error, while minimizing respondent burden and cost.

3. Explain how the survey will be conducted. How will the customers be sampled (if fewer than all customers will be surveyed)? What percentage of customers asked to take the survey will respond? What actions are planned to increase the response rate? (Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to services provided by Web.)

The universe for the sample is defined as ‘users of charts and nautical products relating to navigating in US charted waters’. The selection of the sample frame for this survey is based on many years of reviewing chart users’ lists in relation to how the data is to be used, to develop guidance on how to improve services/products.

Completing a census of users is beyond the budget allocation that is available for the completion of this survey. Therefore NOS has worked with Strategy, Research & Action to develop a sample to give as representative and unbiased data about chart users as possible.

The aim of this survey is to select a representative sample using a stratified sampling technique, in which the chosen sample is forced to contain potential respondents from each of the key segments of the population. Such a sample provides statistical efficiency, as the sampling error is likely to be smaller than using a systematic random sample.

NOS will send self-completion survey forms to 5,000 commercial mariners and 5,000 recreational mariners with a link to the research company’s server, so that the survey can be completed online. This is possible because we have lists of product users’ emails.

The Commercial mariners’ email will mainly be to subscribers to Professional Mariner, as they have the most up to date list of chart users. The Recreational mariners email will mainly be to subscribers to BoatUS, as this is the most up to date list of chart users. The contact lists below are selected to cover all parts of the population of chart users.

We show below if each list will be used in its entirety or sampled, and the expected number to be sampled:

1. Professional Mariner – a sample of 5,000 subscribers from the total of their
   mailing list of 8,000, who are commercial mariners

2. BoatUS – a sample of 5,000 subscribers from the total of their mailing of
   250,000, who are recreational mariners.

A response rate of up to 25% for these surveys is likely to be achieved, based on previous results and because of the following:

• Accuracy of the Email Lists
  NOS has spent time ensuring the accuracy of the email lists and the extent to
  which they are segmented into the relevant users of NOS charts. This includes
  those who are regularly interacting with NOS for all of its services. Use of these lists in the past has also contributed to the high response of up to 25% for the surveys.

• Re-contacting the Target Audience

Past surveys have taken into account the recent requirement referenced in all Data Protection and Codes of Conduct for research*, asking the respondents if they would be willing to take part in future surveys. A high response of 50% has been achieved to this question in the past, indicating that NOS had contacted the correct target audience and this helps to keep the response rate high in future surveys, although it is recognised that not all those responding affirmatively will take part in the future.

• The relationship between NOS and its Chart Users

The on-line self-completion survey has been selected for this survey, because of the relationship between NOS and Chart users.

Past surveys have shown that the respondents:

• Are interested in responding, because the survey was sent by NOS

• Are interested in the subject

• Believe that taking part in the survey will improve the relationship with, and service received from, NOS in the future.

*The Standards are administered for International Research by the Market Research Society (MRS) out of the UK and MRS members are required to use them for projects completed by them internationally.

• The Cover email

The questionnaires will be emailed with a cover email signed by a senior member of NOS staff, which will identify:

• The purpose of the research

• The reasons why NOS would like the recipients to respond

• The ease of completion

• The short time needed to complete the questionnaire

• A reminder that no confidentiality can be assured as a result of participating in the survey, but that respondents can withhold the details of their name and address

• A NOS and research supplier contact name for more information or questions about the survey

• The time scale for return of the questionnaire.

• Delivering the NOS Email

In planning this survey, we have done our best to ensure that there is a high response by taking the following steps:

• Issue an initial email to all the targeted audiences to inform them that a voluntary questionnaire will be emailed shortly and to raise their awareness of its pending arrival.

• Email the questionnaire 7 days later. NOS will inform respondents that the

responses to the collection of information are voluntary. The information will

also state that the questionnaire will take no longer than 10 minutes to complete.

• The OMB Control Number and expiration date (OMB Control No. 0648-0342 – Expires) and a Paperwork Reduction Act Statement will be shown at the end of the on-line questionnaire.

• Email a reminder 7 days after the questionnaire has been emailed reminding the target respondent to return the questionnaire as soon as possible.

Each survey questionnaire will be checked by the research contractor upon receipt. Any inaccurate completions will be discarded and only questionnaires which have been completed accurately and with due consideration will be included in the final analysis.

NOS will implement safeguards throughout the survey production process by asking the research contractors to ensure that survey data are handled to avoid disclosure. This is done by separating the survey responses from classification information, which might identify who has provided the response.

Most common apparent reasons for non-response in these surveys are refusals or non-availability of respondents, likely resulting from the timing of the survey; NOS management is taking care to schedule the survey during periods not including national holidays. Mainly because we have not had obviously skewed response profiles in past surveys, we did not build into the survey, or budget for, any follow-up activities with non-responders.

4. Describe how the results of this survey will be analyzed and used. If the customer population is sampled, what statistical techniques will be used to generalize the results to the entire customer population? Is this survey intended to measure a GPRA performance measure? (If so, please include an excerpt from the appropriate document.)

Analysis and Report Planning

A multivariate analysis will be conducted on the responses and chart user types and other selected classification information, such as affiliations, type of commercial mariner and type of recreational mariner, etc. Means and standard deviations and standard errors for user representation will also be calculated. The survey is not intended to measure a GPRA performance measure. These analyses will be comparable with those of previous surveys, so that similar statistical conclusions can be derived from past survey data as is acceptable statistical practice.

Review of Information Products

NOS will be responsible for the quality of information that they disseminate and will institute appropriate content/subject matter, statistical and methodological review procedures to comply with OMB and agency Information Quality Guidelines.

Releasing Information

NOS will produce a research summary which can be used to release information intended for the general public in line with NOS dissemination policies and procedures. An appendix will describe the procedures that have been used to evaluate the quality of the data, to allow users to interpret results of analyses and to help designers of recurring surveys to focus on improvement efforts.

Data Protection and Disclosure Avoidance for Dissemination

NOS will implement safeguards throughout the survey production process by asking the research contractors to ensure that survey data are handled to avoid disclosure. NOS will ensure that the information that is published or released on request will comply with applicable Federal legislation and regulations.

Survey Documentation

NOS will produce survey documentation that includes those materials needed to analyze data from the survey, as well as the information necessary to replicate and evaluate the survey’s results compared with previous surveys.

Documentation and Release of Public-Use Microdata

If NOS releases microdata to the public it will ensure that it will include documentation clearly describing how the information is constructed and if required will also provide the metadata necessary for users to access and manipulate the data.

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.

The universe for the sample is defined as ‘users of charts and nautical products relating to navigating in US charted waters’. The aim of this survey is to select a representative sample using a stratified sampling technique, in which the chosen sample is forced to contain potential respondents from each of the key segments of the population. Such a sample provides statistical efficiency, as the sampling error is likely to be smaller than using a systematic random sample.

NOS will send self-completion survey forms to 5,000 commercial mariners and 5,000 recreational mariners with a link to the research company’s server, so that the survey can be completed online. This is possible because we have lists of product users’ emails.

The commercial mariners email will mainly be to subscribers of Professional Mariner, as they have the most up to date list of chart users. The recreational mariners email will be mainly to subscribers of BoatUS.

We show below if each list will be used in its entirety or sampled, and the expected number to be sampled:

1. Professional Mariner – a sample of 5,000 subscribers from the total of their
   mailing list of 8,000, who are commercial mariners

2. BoatUS – a sample of 5,000 subscribers from the total of their mailing list of 250,000, who are recreational mariners

2. Describe the procedures for the collection, including: the statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

When the lists have been assembled by NOS management, the names of those who will be contacted will be selected by using a ‘skip interval’, allowing NOS management to go through the lists and select the names on the basis of the skip interval calculated, using the following formula:

k= Size of population = N (Total number in lists provided)

Size of sample required n

In the past use of these lists has provided a high strike rate for contacting chart users. When the sample is selected, a random choice sequence for label selection (by taking a random number and counting the intervals of labels) for the contacts will be adopted, counting through the lists until the sample is full. This has been the procedure used by NOS management in the past that has ensured a good response and meaningful response.

In surveys of this nature, and in the past NOAA surveys completed, usually, 95% confidence limits are used for the accurate interpretation of the data - we are confident of using the 95% level for this survey, as the repeated samplings of the same population of previous surveys identified the properties of the normal distribution. This provides the confidence limits for 95% of the samples selected including the parametric mean.

Therefore, analysis of the data will be subjected to a calculation that will be used to calculate the confidence limits: multiplying the standard error of the mean x the appropriate t-value. This means that there is a 5% probability or a 1 in 20 chance that the result or finding has occurred by chance. This is the lowest acceptable level in most market research business to business projects, such as this survey and will be sufficient for interpreting the data for the guidance NOS needs.

For example, if we received back 195 questionnaires in the survey for one particular chart user type (such as container ship operators), or 195 commercial mariners say that NOAA service has improved in the last 12 months; the following calculations could be made:

• Number Of Observations = 195

• Mean = 9.261460

• Standard Deviation = 0.2278881e-01

• Standard Deviation Of Mean = 0.1631940e-02

Referring to the following table provides the confidence levels that will be used for the survey:

Confidence T T X Sd(Mean) Lower Upper

Value (%) Value Limit Limit

50.000 0.676 0.110279E-02 9.26036 9.26256

75.000 1.154 0.188294E-02 9.25958 9.26334

90.000 1.653 0.269718E-02 9.25876 9.26416

95.000 1.972 0.321862E-02 9.25824 9.26468

99.000 2.601 0.424534E-02 9.25721 9.26571

99.900 3.341 0.545297E-02 9.25601 9.26691

99.990 3.973 0.648365E-02 9.25498 9.26794

99.999 4.536 0.740309E-02 9.25406 9.26886

The table shows the confidence interval for several different significance levels. The first column lists the confidence level (which is 1 - expressed as a percent), the second column lists the t-value, the third column lists the t-value times the standard error, the fourth column lists the lower confidence limit, and the fifth column lists the upper confidence limit.

Therefore, if we use a 95% confidence interval, the row identified by 95.000 in the first column then one can see an interval of (9.25824, 9.26468) from the last two columns.

3. Describe the methods used to maximize response rates and to deal with nonresponse. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield "reliable" data that can be generalized to the universe studied.

Response rates of approximately up to 25% for the surveys are likely to be achieved, based on previous results and because of the following:

Accuracy of the Email List

NOS has spent time ensuring the accuracy of the email list and the extent to which it is segmented into the relevant users of NOS charts. This includes those who are regularly interacting with NOS for all of its services. Use of these lists in the past has also contributed to the high response of 25% for the surveys.

Re-contacting the Target Audience

Past surveys have taken into account the recent requirement made in all Data Protection and Codes of Conduct for research, asking the respondents if they would be willing to take part in future surveys. A high response of 50% has been achieved to this question in the past, indicating that NOS had contacted the correct target audience and this helps to keep the response rate high in future surveys, although it is recognised that not all of those responding affirmatively will take part in the future.

The relationship between NOS and its Chart Users

The online self-completion survey has been selected for this survey, because of the relationship between NOS and chart users.

Past surveys have shown that the respondents:

• Are interested in responding, because the survey was sent by NOS

• Are interested in the subject

• Believe that taking part in the survey will improve the relationship and service
  received from NOS in the future.

The Cover Email

The questionnaires will be emailed with a cover email signed by a senior member of NOS staff, which will identify:

• The purpose of the research

• The reasons why NOS would like the recipients to respond

• The ease of completion

• The short time needed to complete the questionnaire

• A reminder that no confidentiality can be assured as a result of participating in
  the survey, but that respondents can withhold the details of their name and
  address

• A NOS and research supplier contact name for more information or questions
  about the survey

• The time scale for return of the questionnaire.

Administration of the NOS Email

In planning this survey we have done our best to ensure that there is a high response to complete the emailing as follows:

• Issue an initial email to all the targeted audiences to inform them that a
  questionnaire will be emailed shortly and to raise their awareness of its pending
  arrival.

• Email the questionnaire with the covering letter 7 days later.

• Email reminding the target respondent to complete survey as soon as possible.

Each survey questionnaire will be checked by the research contractor upon receipt. Any inaccurate completions will be discarded and only questionnaires which have been completed accurately and with due consideration will be included in the final analysis.

NOS will implement safeguards throughout the survey production process by asking the research contractors to ensure that survey data are handled to avoid disclosure. This is done by separating the survey responses from classification information which would identify who has provided the response.

Most common apparent reasons for non-response in these surveys are refusals or non-availability of respondents, likely resulting from the timing of the survey; NOS management is taking care to schedule the survey during periods not including national holidays.

Mainly because we have not had obviously skewed response profiles in past surveys, we did not build into the survey, or budget for, any follow-up activities with non-responders.

4. Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved OMB must give prior approval.

No tests will be undertaken.

5. Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

NOS has consulted the appointed contractor for the statistical aspects of the design. The Contractor is Robin J Birn, President, Strategy, Research and Action, Inc, 29 Lyon, Newport Coast, CA 92657 Tel: 949-760-3980 Email: [email protected]