Cognitive Interview Protocol

2010 National Census of Ferry Operators

Cognitive Interview Protocol

Kenneth Steve, and Joy Sharp, RITA - BTS

January 28, 2010

Research Questions:

• Graphics and/or navigational aids: Are the respondents able to navigate through the questionnaire easily? Are there any graphical elements that consistently distract respondents?

• Question Wording: Are the respondents able to understand the verbiage used in each question? Does the respondent understand what information is being asked for in each question?

• Respondent Selection: How are operations deciding who will respond to the survey?

• Respondent Burden: What steps are required of the respondent to gather the information necessary to complete the census questionnaire? Are any of the data elements being asked for extremely difficult for the respondent to provide? If so, how would they respond?

• Pre-populated Data: Does pre-populating data fields it reduce respondent burden? How would the respondent go about verifying that the information is correct?

Expected Length of Interview: 60 minutes

Required Materials:

• 1 blank NCFO questionnaire

• 1 NCFO questionnaire with pre-populated data

• Consent forms

• Digital recorder

• NCFO instruction booklet

Introduction:

• Background: Federal law requires that the Bureau of Transportation Statistics (BTS) establish and maintain a national ferry database containing current information regarding routes, vessels, passengers and vehicles carried, and such other information as the Secretary considers useful. BTS has conducted the NCFO every other year since 2006. The data collected by the NCFO allows lawmakers to have a clear picture of ferry operations in the U.S. and its possessions (i.e., the 50 states, the District of Columbia, Puerto Rico, the U.S. Virgin Islands, Guam, and the remaining territories, and commonwealths). The 2010 NCFO is being conducted in order to update that picture.

• Purpose: The purpose of the interview is to gather information that will allow BTS to better understand the processes used to answer the census questions, to get feedback on current form, to make changes and make it easier to complete. We are not necessarily interested in actual answers to questions, but rather how you would go about getting the answers. The interview is simply a means of pre-testing the census form to identify problems so they may be corrected prior to conducting the actual census. There are no wrong answers. Any feedback you can provide is useful.

• Meeting structure: I will begin by asking a few general questions about your operation. After that I lead you through a brief exercise to get you used to the interview process. The process is very similar to a normal conversation, where I ask you a series of questions and you respond. However, instead of simply answering the questions, I would like you to verbalize your thoughts as you are coming up with the answer. Again, there are no wrong answers. We are interested in learning about what naturally comes to mind when you are asked the question.

Permission:

Since I will be facilitating the interview, I will be unable to keep good notes. I would like to audio tape the interview so that I can study your responses in more detail after the interview is complete. Once I have finished my analyses, the recording will be destroyed. As with all research conducted by BTS, all information you provide will be kept strictly confidential. At no time will this recording be shared with anyone outside BTS. Is it OK to record this interview session? (Present consent form).

General Probes:

• What were you thinking?

• Can you tell me more about that?

• You answered "..." because...?

• I want to make sure I understand, can you explain that again?

• In your own words, what was that question asking?

• Can you tell me how you arrived at that answer? What did you include/exclude?

General Questions:

• If this census form was sent to your operation in the mail, who would take responsibility for filling it out? Why?

• What is you/that person’s role within the company? How long have you/they been with the company?

• Would you/that person have access to information about the vessels inn the fleet? Where is information about the vessels in your fleet kept?

• Would you/that person have access to information about the routes and schedules your company services? How is this information stored and maintained?

Warm Up:

I’m going to begin the next part of the interview now, so I would like to try a couple warm up exercises to get you used to the process of thinking out loud when responding to my questions. I’m going to ask you a question and instead of just giving the answer, I would like you to tell me what you are thinking as you are coming up with the answer.

• Please tell me how many windows there are in your house. Prompt as needed to elicit think aloud.

• Please tell me how many closets there are in your house. Coach as necessary.

Questionnaire Review:

Now instead of asking questions, I’m going to give you a copy of the census form. Please review the form as you would if you were to have received it in the mail and were going to complete the form. Again, it is there are no wrong answers. I’ only concerned with learning
how you would go about filling out the form. Please verbalize your thoughts as you go.

• Hand blank form to respondent.

• Allow respondent to go over the form without prompting them to focus on any given piece. Remind them to verbalize their thoughts as needed.

• Once the respondent has reviewed the entire form, probe the following specific areas:

Page 3:

What would they do if the information on the form is correct?

How would they correct information that is incorrect?

Page 4:

What is the difference between direct and contracted public operation?

What is the difference between a contracted public and contracted private operation?

What is the difference between a contracted private and a commercial operation?

Where the definitions useful? How could they be improved?

Ask the following probes as needed based on the initial review:

Page 5:

How would you verify the information on this page?

What is a CEU?

Have you ever heard of the terms “Lane Feet” or Lane Meters”?

If information on this page was wrong, what would you do to correct it?

Page 6:

How would you estimate a vessels operating speed?

If information on this page was wrong, what would you do to correct it?

Pages 7 & 8:

If the information on this page was wrong, what would you do to correct it?

Page 9:

How would you estimate the length of a route segment?

Does each of the route segments stick to a marked channel?

IF NOT: Does each route segment operate in a straight line between terminals?

If information on this page was wrong, what would you do to correct it?

Page 10:

How are you estimating the total passenger boardings for each route segment?

How would you indicate the days of the week for a peak passenger period?

How would you estimate the average number of passenger boardings for a peak period?

If the information on this page was wrong, what would you do to correct it?

Page 11:

How are you estimating the total vehicle boardings for each route segment?

How would you indicate the days of the week for a peak vehicle period?

How would you estimate the average number of vehicle boardings for a peak period?

If the information on this page was wrong, what would you do to correct it?

Wrap up:

Are there any other thoughts, comments or concerns you’d like to share about the census form?

That completes our interview session. On behalf of the US Department of Transportation and the Bureau of Transportation Statistics, I would like to sincerely that you for taking part in this important process.

2010 NCFO: Cognitive Interview Protocol Page 3 of 3