Cognitive Interview Protocol

Package Testing

Protocol for 2015 Updated National Survey of College Graduates Mail packages letters

Introduction:

Hello, my name is _______________. I work for the [fill contractor] on behalf of the Census Bureau. Thanks for agreeing to help us today. Let me start by telling you a little about what we will be doing today. Every two years the U.S. Census Bureau conducts the National Survey of College Graduates (NSCG) on behalf of the National Science Foundation which is designed to provide data on the number and characteristics of individuals with a college degree, or higher, living in the United States. In order to improve the next NSCG, we are asking you to look at some of the mailing materials that you would get in the mail before and during the 2017 NSCG. We will be looking at about 10 mailing packages. I’ll explain each one to you as we go along. I’m going to ask you to look at the envelopes and read the letters, and then ask you some questions about them: what they mean to you, and what your reactions to them are.

There are no right or wrong answers. We are interested in what people think about these mailing materials.

Permission to Tape-Record:

So I don’t have to rely on my memory later on, I’d like to tape record this interview. That way, I can focus today on what you’re saying rather than having to concentrate on taking notes. Is that ok with you? [Hand respondent consent form.] Please read this over and sign it. I want you to know that your responses will be kept strictly confidential and will only be seen by Census employees. Your participation in this study is completely voluntary and you can decline to answer any particular question.

Think-out-loud Instructions:

As I said before, we are interested in your thoughts on these envelopes and letters. What I would like you to do is to read each one as I give it to you. So I know what you are thinking and how you are reacting to it, as you read it, I’d like you to tell me everything that’s going through your mind. Instead of thinking to yourself, I’d like you to think out loud. From time to time, I may ask you what you are thinking to remind you to do this.

I find it helps people to get the idea of thinking aloud while they are answering a question if they do a practice question first. Let’s begin with a practice question. Remember to think aloud as you answer.

How many windows are there in the house or apartment where you live?

_____________ WINDOWS

[IF NEEDED:] Try to visualize the place where you live, and think about how many windows there are in that place. As you count up the windows, tell me what you are seeing and thinking about.

PROBES:

• How did you come up with that answer?

• Tell me more about that. Why did you say [ANSWER]?

• I noticed that you hesitated. Tell me what you were thinking.

Great! We are ready to get started, do you have any questions before we begin?

TURN ON TAPE RECORDER

General Probes:

What are you looking at?

What are you thinking?

INITIAL LETTER:

SET-UP. The first piece of mail you would receive is this letter I am going to show you now. Please look at this letter, and remember to think aloud as you go through it.

Standard Probes: (when R indicates he/she is done reading.)

1. BY OBSERVATION: Does R look at both sides of the envelope? Read it?

2. Do you think this envelope would stand out in your mail? Would you notice it? Would you open it?

3. How much of the letter did you read? Where did you stop?

4. Was there anything confusing in the letter?

5. What do you think the main point of the letter was? What is the main point of the first paragraph? The second paragraph? Do you think these paragraphs would encourage you to complete the NSCG?

6. Was there any part of the letter you spent more time on than other parts?

7. What do you think you would do after receiving this letter?

8. Did the letter mention any reasons for doing the NSCG? What were they? Did you feel the reasons were valid? [If they say more than one reason then ask “which reason would you say is the most important”]

9. Was there anything you thought was missing from the letter?

REMINDER LETTERS:

SET-UP: After you get the initial letter, if you haven’t completed your online survey, we will send you letters to remind you to complete the survey. This is one of the reminder letters. Please look at the envelope and letter and remember to think aloud as you go through it.

1. BY OBSERVATION: Does R look at both sides of the envelope? Read it?

2. How much of the letter did you read? Where did you stop?

3. Was there any part of the letter you spent more time on than other parts?

4. What do you think the main point of the letter was?

5. What do you think you would do after receiving this letter?

6. Did you notice any new information in this letter that you hadn’t seen yet?

7. What is the main point of the first paragraph? The second paragraph? Do you think these paragraphs would encourage you to complete the NSCG?

DEBRIEFING:

I just have a few more questions for you.

1. In general, would you say you were likely or not very likely to read these materials if they came in the mail to you? If no, is there anything we could add that would make you more likely to read them?

2. If you are not likely to respond to surveys, is there anything we could add to the envelope or the letter that could convince you to respond?

3. Let’s say that you completed your survey online, and then got another letter reminding you to respond to the NSCG. What would your reaction be?

4. Do you think that the language of the letters and postcards is too difficult? Too easy? About just right? Are there any specific words and phrases you think that some people might not be familiar with?

5. Did the messages in the letters bring up any issues in your mind that some people might find sensitive or uncomfortable?

6. Was there anything else that we haven’t talked about that you would like to mention?

Thank you for participating in this study. Your feedback has been very helpful.

TURN OFF TAPE RECORDER AND GIVE INCENTIVE. HAVE R SIGN THE PAYMENT VOUCHER.

Here is your money for helping me today. I need you to sign this payment voucher so the government knows that I gave you the money.