Protocol

SmaRT Testing of Decennial Mail Materials

RESPONDENT ID #: _________________________ DATE: ____ / ____ / _____

INTERVIEWER’S NAME: _________________________________________________

START TIME: ______: ______ AM / PM TEXT MESSAGE VERSION: _____________

LOCATION: _____________

Introduction

Hello, my name is ________________, and I work for the Census Bureau. Thank you for agreeing to help us today. Let me start by telling you a little about what we will be doing today.

Every year, the Census Bureau conducts a number of surveys. Today we are going to look at some of the envelopes, letters, and postcards that you would receive in the mail, and the questionnaire that you would answer online if you were completing the Census.

[If needed: Because these are example mailings, they may not look quite as nice as they would if you received them in the mail.] We will go through four different pieces of mail one by one. After you’ve finished with one piece of mail, I will have you look at the next.

As we go through these mailings, I’m going to ask you to treat each one as you would if it came to your home. Please go through these as you would at home (as if I were not [here/there]), but I’d like you to say out loud what you are thinking as you look at the mailings.

I’m also going to ask you some questions about the mailings and about your reactions to them. There are no right or wrong answers. I really want to hear your opinions and reactions to these pieces of mail, so don’t hesitate to speak up whenever something is unclear. Please feel free to be honest.

[IF OBSERVER PRESENT]: I want to let you know that this interview is being observed by other Census Bureau staff. They are observing the interview because they are also interested in how the mailings work for people like you.

Do you have any questions about what we’ll be doing?

Informed Consent

• IF IN PERSON [HAND RESPONDENT COPY OF THE CONSENT FORM.]

Before we start, I would like you to read over the document in front of you. This document explains a little bit about this interview and provides information about your rights as a participant, that your participation is optional, all information you provide is confidential, and we won’t use your name in any report. It also asks for your permission to have this session audio recorded and for your screen to be recorded so that I can concentrate on what you are saying rather than having to take extensive notes. Please ask me any questions you have about this document. Once you have finished reading the document, please sign and date it.

• [RESPONDENT READS FORM THEN SIGNS AND DATES FORM]

• [INTERVIEWER SIGNS AND DATES FORM]

• [IF RESPONDENT PROVIDES CONSENT TO HAVE THE SESSION RECORDED]: I will now turn on the recorder.

• [TURN ON RECORDER]

Thank you for signing the consent form. I will also sign this form to show that I agree to everything in the form.

IF REMOTE

[Participant was emailed consent form]. We emailed you information about the session today and your rights as a participant. This document explains a little bit about this interview and provides information about your rights as a participant, that your participation is optional, all information you provide is confidential, and we won’t use your name in any report. It also asks for your permission to have this session audio recorded and for your screen to be recorded so that I can concentrate on what you are saying rather than having to take extensive notes. Please ask me any questions you have about this document.

I just need to capture your consent orally. Only those of us connected with the project will review the recording and it will be used solely for research purposes. Your name will not be associated with the recording or any of the other data collected during the session. Do you give your consent?

• [IF RESPONDENT PROVIDES CONSENT TO HAVE THE SESSION RECORDED]: I will now turn on the recorder.

• [TURN ON RECORDER]

Think Aloud

I’d like you to answer the questions as you would if you were at home without me there, but with one big difference. What I would like you to do is tell me what you are thinking and feeling as you come up with your answers. 

Let’s do a practice question before we start: Please think aloud as you answer the question, how many windows are in your home?

[GIVE FEEDBACK ON THE THINK ALOUD THEY DID]

IF REMOTE

Camera

So, just like we practiced in the tech check – I am going to need you to go ahead and turn your camera on by clicking on the camera icon.

[If able to see P] Okay great I can see you. Now we can get started with the task.

[If unable to see P will need to troubleshoot it]

Mailing 1 - initial letter: Okay, let’s get started.

[TA NOTE: Mailing 1 has a separate FAQ sheet that is included in the mailing]

[Remote] Please open the priority mail envelope. There are a number of pieces of mail in there so be sure you are in a place to lay them out. We will be referring to each one twice in this session. You will see several envelopes inside. Please find the one labeled Mailing 1 and open it. There should be a letter with a handwritten note at the top. What does it say? [Mailing 1]

[In person] [Hand the participant Mailing 1]

Imagine you received this piece of mail at your house. Take a look at it as if I were not [here/there] and remember to think out loud as you do. You can also let me know if anything stands out to you.

[If participant reads the letter aloud.] You do not have to read aloud unless that is what you would do at home.

OBSERVE (DO NOT READ TO PARTICIPANT):

1. What does P look at (or comment on) first?

___ Letter ___ FAQ

2. Does P need to be told to look at the letter?

___ Yes ___ No ___ Not sure

3. Does P comment on two-step instruction?

___ Yes ___ No ___ Not sure

4. Does P comment on “required by law”?

___ Yes ___ No ___ Not sure

5. Does P comment on “send you a paper questionnaire”?

___ Yes ___ No ___ Not sure

6. Does P comment on why their response matters?

___ Yes ___ No ___ Not sure

7. Does P comment on FAQs?

___ Yes ___ No ___ Not sure

8. Does P comment on 10-minute survey length?

___ Yes ___ No ___ Not sure

9. Does P comment on bilingual aspect?

___ Yes ___ No ___ Not sure

10. Does P spontaneously mention wanting to know when it’s due?

___ Yes ___ No ___ Not sure

11. Does P spontaneously comment about whether it’s confidential?

___ Yes ___ No ___ Not sure

Note any questions P asked/comments P made about this mailing:

Other notes:

[If P does not provide enough information during think aloud] Do you have any reactions or any immediate thoughts about what you just read? 

Thank you. You can put that letter aside with the envelope and FAQ. We will come back to it later.

Mailing 2 - reminder letter:

[Remote] Please open the envelope labeled Mailing 2. There should be a letter with a handwritten note at the top. What does it say? [Mailing 2]

[In person] [Hand the participant Mailing 2]

A few days after you get the first letter, you would receive this reminder letter. Imagine you received this piece of mail at your house. Take a look at it as if I were not [here/there] and remember to think out loud as you do. You can also let me know if anything stands out to you.

[If participant reads the letter aloud.] You do not have to read aloud unless that is what you would do at home.

OBSERVE (DO NOT READ TO PARTICIPANT):

1. Does P comment on two-step instruction?

___ Yes ___ No ___ Not sure

2. Does P comment on “required by law”?

___ Yes ___ No ___ Not sure

3. Does P comment on “answers are confidential”?

___ Yes ___ No ___ Not sure

4. Does P comment on “send you a paper questionnaire”?

___ Yes ___ No ___ Not sure

5. Does P comment on why their response matters?

___ Yes ___ No ___ Not sure

6. Does P comment on bilingual aspect?

___ Yes ___ No ___ Not sure

Note any questions P asked/comments P made about this mailing:

Other notes:

[If P does not provide enough information during think aloud] Do you have any reactions or any immediate thoughts about what you just read? 

Thank you. You can put that letter aside. We will come back to it later.

1^st Text message:

[Remote] I’m going to share my screen and show you a text message you might receive on your phone.

[In person] I’m going to show you a text message that you might receive on your phone. [Show on laptop].

After you receive the second letter, if you had not completed the survey, you would receive this text message. Take a look at as if I were not [here/there] and remember to think out loud as you do. You can also let me know if anything stands out to you.

1. Does P comment on URL/no URL?

___ Yes ___ No ___ Not sure

Note any questions P asked/comments P made about this text message:

Other notes:

[If P does not provide enough information during think aloud] Do you have any reactions or any immediate thoughts about what you just read? 

Mailing 3 - postcard:

[Remote] In the package/priority mailer you received, there should also be a postcard. Please take that out now.

[In person] [Hand the participant Mailing 3]

A few days after you receive the reminder letter, you would receive this postcard. Imagine you received this piece of mail at your house. Take a look at it as if I were not [here/there] and remember to think out loud as you do. You can also let me know if anything stands out to you.

[If participant reads the letter aloud.] You do not have to read aloud unless that is what you would do at home.

OBSERVE (DO NOT READ TO PARTICIPANT):

1. Does P comment on format (postcard v letter)?

___ Yes ___ No ___ Not sure

2. Does P comment on tone/”several requests”/bold text “now is the time to respond?”

___ Yes ___ No ___ Not sure

3. Does P comment on response “required by law”?

___ Yes ___ No ___ Not sure

4. Does P comment on absence of Census ID or location under barcode?

___ Yes ___ No ___ Not sure

5. Does P comment on “send you a paper questionnaire”?

___ Yes ___ No ___ Not sure

6. Does P comment on bilingual aspect?

___ Yes ___ No ___ Not sure

Note any questions P asked/comments P made about this mailing:

Other notes:

[If P does not provide enough information during think aloud] Do you have any reactions or any immediate thoughts about what you just read? 

Thank you. You can put that postcard aside. We will come back to it later.

Mailing 4 - final letter:

[TA NOTE: Mailing 4 has a separate FAQ sheet that is included in the mailing and questionnaire]

[Remote] Please open the envelope labeled Mailing 4. There should be a letter with a handwritten note at the top. What does it say? [Mailing 4]

[In person] [Hand the participant Mailing 4 in envelope]

If you had not completed the survey, about one week after the postcard you would receive this final mailing. Imagine you received this piece of mail at your house. Take a look at as if I were not [here/there] and remember to think out loud as you do. You can also let me know if anything stands out to you.

[If participant reads the letter aloud.] You do not have to read aloud unless that is what you would do at home.

OBSERVE (DO NOT READ TO PARTICIPANT):

1. What does P look at (or comment on) first?

___ Letter ___ FAQ ___ Paper questionnaire

2. Does P comment on FAQs?

___ Yes ___ No ___ Not sure

3. Does P comment on text “choose ONE way to respond”?

___ Yes ___ No ___ Not sure

4. Does P comment paper questionnaire enclosure?

___ Yes ___ No ___ Not sure

5. Does P comment on “required by law”?

___ Yes ___ No ___ Not sure

6. Does P comment on interviewer collecting data in person?

___ Yes ___ No ___ Not sure

7. Does P comment on due date in either location?

___ Yes ___ No ___ Not sure

8. Does P comment on bilingual aspect?

___ Yes ___ No ___ Not sure

Note any questions P asked/comments P made about this mailing:

Other notes:

[If P does not provide enough information during think aloud] Do you have any reactions or any immediate thoughts about what you just read? 

Thank you. You can put that letter aside. We will come back to it later.

2^nd Text message:

[Remote] I’m going to share my screen again and show you a text message you might receive on your phone.

[In person] I’m going to show you a text message that you might receive on your phone. [Show on laptop].

After you receive the final letter, if you had not completed the survey, you would receive this text message. Take a look at as if I were not [here/there] and remember to think out loud as you do. You can also let me know if anything stands out to you.

1. Does P comment on URL/no URL?

___ Yes ___ No ___ Not sure

Note any questions P asked/comments P made about this text message:

Other notes:

[If P does not provide enough information during think aloud] Do you have any reactions or any immediate thoughts about what you just read? 

Probes

Thank you. Now I have a few questions I’d like to ask you about the mailings. We’ll go through them again one at a time.

• Only ask probe if not mentioned during think aloud.

• Take back each mailing after respondent is finished with it—do not let them compare the mailings

• Hand materials to respondents SIDEWAYS, so that it is neither face up or face down.

Mailing 1 - initial letter:

[TA NOTE: Mailing 1 has a separate FAQ sheet that is included in the mailing]

Research Questions:

1. How do participants react to minimalist design?

2. Do they understand the purpose of the letter?

3. Do they understand that they are required to complete the census?

[Remote] Please take a look at the first mailing that we had you look at. It should be labeled as Mailing 1.

[In person] This is the first mailing you received. [Give respondent mailing with letter on top.]

1. If needed: [If in person] When you first opened the letter, what did you notice about it?

[If remote] When you first opened the letter, what did you notice about it?

1. What about the FAQs? Did you notice them? If so, did you read through them? [If P has not read through them, ask them to]

2. Are there other questions you think should be included in the FAQs?

2. What do you think the main point of this mailing is? [Do they understand that they have to complete the census?]

3. What would you have done next after reading it? (If needed: Tell me more about that.)

4. [If not mentioned during think aloud] What are your thoughts about the length of the survey?

5. [If not mentioned during think aloud] Did you notice anything in the letter about why you should complete the survey? Tell me more that.

6. [If not mentioned during think aloud] Do you have any feedback about the instructions to respond?

7. [If any observations marked as yes above] I noticed you/you commented on…

Mailing 2 - first reminder:

Thank you. [Take back previous mailing.]

[Remote] Please take a look at the second mailing that we had you look at. It should be labeled as Mailing 2.

[In person] This is the second mailing you received. [Give respondent mailing.]

1. If needed: [If in person] When you first opened the letter, what did you notice about it?

[If remote] When you first opened the letter, what did you notice about it?

2. What do you think the main point of this mailing is?

3. Did this seem the same as the first mailing, or did it seem different? (If needed: Tell me more about that.)

4. What would you have done next after reading it? (If needed: Tell me more about that.)

5. If you hadn’t responded already, after getting this letter, do you think you would respond to the survey?

6. What does the word “confidential” mean to you in this sentence? What if it said, “completely confidential?” What if it said, “your answers are protected”?

7. [If any observations marked as yes above] I noticed you/you commented on…

8. Mailing 3 - postcard:

Thank you. [Take back previous mailing.]

[Remote] Please take a look at the postcard.

[In person] This is the third mailing you received. [Give respondent mailing.]

1. If needed: When I first gave you this, what did you notice about it?

[If remote] When you first saw this, what did you notice about it?

2. What do you think the main point of this mailing is?

3. Did this seem the same as any of the previous mailings you saw, or did it seem different? (If needed: Tell me more about that.)

4. What would you have done next after reading it? (If needed: Tell me more about that.)

5. [If not mentioned] Do you have any thoughts about the tone of the postcard? [If needed] If you hadn’t responded to the census at this point, how would this postcard make you feel?

6. [If any observations marked as yes above] I noticed you/you commented on…

Mailing 4 -final:

Thank you. [Take back previous mailing.]

[Remote] Please take a look at the fourth mailing that we had you look at. It should be labeled as Mailing 4.

[In person] This is the fourth and final mailing you received. [Give respondent mailing.]

1. If needed: [In person] When I first gave you this, what did you notice about it?

[If remote] When you first opened the letter, what did you notice about it?

2. What do you think the main point of this mailing is?

3. Did this seem the same as any of the previous mailings you saw, or did it seem different? (If needed: Tell me more about that.)

4. What would you have done next after reading it? (If needed: Tell me more about that.)

5. This mailing came with the paper questionnaire for you to complete. What do you think about that?

6. Do you think you can complete the questionnaire over the phone?

7. [If remote, highlight sentence] [If in person, point to sentence] What does the phrase “Choose ONE way to respond” mean to you?

8. [If not mentioned during think aloud] Did you notice the date on this letter? What did you think about that?

9. Did you notice the due date on the envelope? What do you think about that?

10. [If any observations marked as yes above] I noticed you/you commented on…

Text messages:

Thank you. [Take back previous mailing.]

[Remote] I’m going to share my screen again and show you those text messages as well as some other options.

[In person] I’m going to show you those text messages as well as some other options. [Show on laptop].

1. [Show version of reminder 1 text message that P saw] This is the first reminder text message that you saw earlier. Do you have any thoughts that you didn’t already share?

2. [If version 1] Would you click on the link?

3. [Show alternative] What if this were the text message? What are your thoughts?

4. Do you have a preference between these two?

5. [Show version of reminder 2 text message that P saw] This is the second reminder text message that you saw earlier. Do you have any thoughts that you didn’t already share?

6. [If version 1] Would you click on the link?

7. [Show alternative] What if this were the text message? What are your thoughts?

8. Do you have a preference between these two?

Well, that’s all the questions I have about the letters. Thank you so much for your feedback. [If remote] You can just discard the materials. You do NOT need to send in that paper questionnaire.