2019 ACS Due Date Test Revised Mailing Materials Cognitive Interviews

Cognitive Testing of ACS Due Date Materials

PARTICIPANT ID #: _________________________ DATE: ____ / ____ / _____

INTERVIEWER’S NAME: _________________________________________________

START TIME: ______: ______ AM / PM

Introduction

DO NOT MENTION DUE DATES in materials until probing.

Hello, my name is ________________, and I work for the Census Bureau. Thank you for agreeing to help us today. Let me start by telling you a little about what we will be doing today.

Every year, the Census Bureau conducts a number of surveys. Today we’re going to look at some of the items that might be mailed to you if you were chosen to participate in one of those surveys, the American Community Survey. We will go through some different pieces of mail one by one. I’m also going to ask you some questions about some of the mailings and about your reactions to them. There are no right or wrong answers. I really want to hear your opinions and reactions to these pieces of mail, so don’t hesitate to speak up whenever something is unclear. Please feel free to be honest.

[IF OBSERVER PRESENT]: I want to let you know that this interview is being observed by other Census Bureau staff. They are observing the interview because they are also interested in how Census Bureau mailings work for people like you.

Do you have any questions about what we’ll be doing?

Informed Consent

• [HAND PARTICIPANT COPY OF THE CONSENT FORM.]

Before we start, I would like you to read over the document in front of you. This document explains a little bit about this interview and provides information about your rights as a participant, such as that your participation is optional, all information you provide is confidential, and we won’t use your name in any report. It also asks for your permission to have this session audio recorded so that I can concentrate on what you are saying rather than having to take extensive notes. Please ask me any questions you have about this document. Once you have finished reading the document, please sign and date it.

• [PARTICIPANT READS FORM THEN SIGNS AND DATES FORM]

Thank you for signing the consent form. I will also sign this form to show that I agree to everything in the form.

• [INTERVIEWER SIGNS AND DATES FORM]

• [IF PARTICIPANT PROVIDES CONSENT TO HAVE THE SESSION RECORDED]: I will now turn on the recorder.

• [TURN ON RECORDER]

Think-Aloud

While you are looking at the materials I give you, tell me what information you are seeing, any questions that come to your mind, and reactions you’re having.

No practice think aloud. Encourage participants to think aloud as needed throughout interview, especially with first few mailing pieces:

• What are you looking at?

• What are you thinking?

• Remember to tell me what you are thinking about as you look at these mailings.

Set 1: Observation

Research Questions:

1. Do participants easily notice the due date messaging, where do they notice it, do they feel it is repetitive or helpful? Does it add a sense of urgency for them? As applicable: Do they easily notice the text about the interviewer visit or being added or removed from a schedule?

2. Do they understand each version’s modified text as we intended? What does it mean to be added/ removed from a schedule?

3. How do respondents interpret the message about an interviewer coming to their home? Do they think that they “may come” or that “they definitely will come”?

4. For modified Remove version vs simple Remove version, is one version clearer to participants than the other? Which version, if any is more likely to result in a response?

Okay, let’s get started. Pretend you were selected to participate in the American Community Survey. You would receive multiple pieces of mail from the Census Bureau about this survey. I am going to hand you each piece of mail, so you can see what they look like, but I don’t want you to open them, I will let you know what each one says.[Place the mailing in front of respondents, but don’t let open it].

[Hold up first mailing package place in front of them but do not let open.] First, about a month ago, you would have received this letter that tells you that your household has been selected to participate in the survey and tell you where you can go online to respond [After they have a minute or two to look take back]. [Hold up pressure seal reminder, place in front don’t let open] Then about a week later, you would receive this reminder to respond to the survey, just in case you had not responded or seen the first letter [Take back after a minute or two]. [Hold up Questionnaire mailing, hand to them] If you hadn’t responded to the survey online within two weeks of the reminder, you would receive a paper survey in case you don’t want to respond online, however you can still also respond online if you prefer. [hand 4^th mailing piece] A few days later, you would receive this reminder to respond to the survey. If you do not respond to the survey within the next two weeks, online or by mail, you would receive this letter. We are going to focus on this last letter. [Hand version (Neutral, Add, Remove) of 5^th mailing participant has been randomly assigned. It will be in a pressure seal with no due date]

Note non-verbal and verbal reactions to the first four mailings:

• Make observation notes below (see questions in red text). Note any nonverbal reactions.

• Let the participant spend no more than 5 minutes looking at each version of the 5^th mailing.

• Take back each mailing after participant is finished with it—do not let them compare the mailings to each other until specified.

Mailing 5– letter (Randomly assign “Neutral, Add, or Remove” Versions): Please treat this letter as you would if you were receiving this piece of mail at your home. Remember to tell me what you are thinking.

OBSERVE (DO NOT READ TO PARTICIPANT):

1. Does R comment on the due date in the letter?

___ Yes ___ No ___ Not sure

2. Does R comment on the part about a visit from a Census Bureau interviewer?

___ Yes ___ No ___ Not sure

3. Does R comment on requested response?

___ Yes ___ No ___ Not sure

If yes, where? (Can be more than one place): __________________________________

4. Does R read letter; if so, how closely?

___ Did not look ___ Glanced ___ Skimmed ___ Read closely

5. At the time of the interview how many days is it until the due date the respondent sees?

____Days

Note any questions R asked/comments R made about this mailing:

Other notes:

Probe

Thank you. Now I have a few questions I’d like to ask you about the mailing.

• Only ask probe if not mentioned during think aloud.

1. Could you take this highlighter and mark what sticks out the most to you about this letter? If nothing sticks out, that is okay too.

2. Is there any part of the letter that is confusing or unclear?

3. What would you have done next after reading it? [If needed] Tell me more about that.

4. [If not mentioned yet: point to paragraph after box] What does this text mean to you? [If needed] Did you notice the due date? Would it make you more likely to respond right away?

5. [If needed] How do you feel about the possibility of an interviewer coming to your house?

Version specific probes:

• “Add” Version: [If needed] What does “add you to our schedule for a visit” mean to you in this paragraph?

• “Remove” Version: [If needed] What does “remove you from our schedule for a visit” mean to you in this paragraph?

Additional Letter 1:

[Take back the first letter, and hand participant the next letter in the folder] This next letter is a different version of the letter you just saw. We have highlighted the part of the letter that is different.

1. Now looking at this letter, please tell me what comes to your mind as you read it. [Let them read and look for up to 5 minutes]

2. What does the highlighted text mean to you?

3. What would you do after reading this letter?

“Add” version: [If needed] What does “add you to our schedule for a visit” mean to you in this paragraph? Does a “schedule” make sense to you in this sentence? [If they say no] What if we had said “add you to our list for a visit”, is that clearer?

“Remove” version: [If needed] What does “remove you from our schedule for a visit” mean to you in this paragraph?

Additional Letter 2:

[Take back the previous letter, and hand participant the next letter in the folder] This next letter is also a different version of the letters you have seen. We have highlighted the part of the letter that is different.

1. Now looking at this letter please tell me what comes to your mind as you read it? [Let them read and look for up to 5 minutes]

2. What would you do after reading this letter?

3. What does the highlighted text mean to you?

V3: [If needed] What does “add you to our schedule for a visit” mean to you in this paragraph?

V4: [If needed] What does “remove you from our schedule for a visit” mean to you in this paragraph?

“Modified Remove” Letter (Version 4.1)

[Take back previous letter and hand them the sheet with just text from “Remove” and “Modified remove”] Thank you for your feedback. Next we are going to look at another paragraph that could be included in these letters.

1. Now looking at this text, can you tell me what the highlighted paragraph means? [Let them read up to 5 minutes]

2. Which of these two paragraph is clearer to you, if any?

3. Which paragraph do you prefer, if any?

4. [Only ask if this has not been mentioned before about any letters] Does a “schedule” make sense to you in these letters? [If they say no] What if we had said “list for a visit”, is that clearer?

Pressure Seal with Due Date

[Take back paper with two versions] Thank you. We are almost done with the interview. I just have one more thing to show you. [Hand pressure sealed in envelope with due date]

1. Please tell me what you would be thinking if you had received this in the mail.

2. [If needed] Do you think you would be more likely to open this envelope than the first envelope? [Hold up opened pressure seal from letter 1.]
   
   
   

3. [If needed] Did you notice the due date on the outside of the envelope? What do you think about that?

Those are all the questions I have for you today, do you have anything else you would like to share?

Thank you again for your time and feedback. [Turn off recorder and have them sign voucher, hand cash incentive].

7