Decennial Instrument Protocol

SmaRT Testing of Decennial Mail Materials – Instrument Testing

RESPONDENT ID #: _________________________ DATE: ____ / ____ / _____

INTERVIEWER’S NAME: _________________________________________________

START TIME: ______: ______ AM/PM VERSION of Vacancy Question 1AB or 2AB: ________

Session start

For the remainder of the session you’ll answer the 2024 National Census Survey that you read about in the letters. Then we will have you answer a few questions about your experience with the survey. I want you to respond to it naturally but just like you have been doing so far, remember to think aloud as you do it. If you fall silent, I may remind you to think aloud by saying “keep talking” or “what are you thinking?”

[HAVE RESPONDENT COMPLETE THE ONLINE QUALTRICS DECENNIAL QUESTIONNAIRE;

Please take a look at Mailing 4. Please use this mailing to respond to the survey like you would if you were at home.

Make note if they had any difficulty with the address screen.

Make note of if the address standardizer found their address [if all capital letters in address that is returned to them] and how they answered the question.

Make note of how they answered the whole/partial question and if they said they did not list everyone how they responded to the forwarding the survey request

Make note of which vacancy question they got– 1A/B or 2A/B, and how they answered

Make note of how they responded to the time capsule.

Now after you hit submit, you’ll receive an email confirmation. You can look at your email and let me know if you have any feedback on that email.

I’m going to put a link in the chat for you to click on.

[THIS LINK CONTAINS THE SATISFACTION QUESTIONS, THEN DEBRIEFING, THEN KNOWLEDGE CHECK QUESTIONS]

Thank you. You can go to the next page. And I’m now going to have you answer some satisfaction questions about the survey you just completed.

Debriefing

Now I just have a few more questions for you about the survey. Please go to the next page.

[Show Finding the ID] On some of the mailing notifications, the login ID is not in the body of the text. Please describe how you figured out what ID to enter.

[Show Census ID] The ID you entered was 12 digits. For security reasons, sometimes IDs need to include letters and numbers like the example on the left. Sometimes to meet security requirements, IDs need to have more digits, like the 16 digit ID on the right side. Do you have any preference for IDs? That is, would you prefer to enter an ID with letters and numbers or a longer ID with just numbers, or does not matter to you? Please tell me more about your preference.

[If count discrepancy screen is triggered] Take a look at this screen. Any thoughts when you saw this?

[Show respondent the Time Capsule question screen] Take a look at this screen. Any thoughts when you saw this?

Is this what you were expecting based on the envelope or were you expecting something else?

We have three different draft questions. I’m going to show you two other versions. Go to the next page please.

Please read through these messages. The first one is the one you saw and the other two are different versions. Is there one of them that you like better than the others? Which one? Can you tell me more about that?

Go to the next page please.

If you selected Help you would see this text. Please read through this message and share your thoughts.

Do you think this text should be in a help pop-up window or beneath the question itself?

[Show respondent the whole/partial question screen] Take a look at this screen. I believe you answered ______. Can you tell me more about that? [If needed] Who were you including when you answered that question? Did you think of including or excluding anyone [else]? What did you think about that screen?

[Show respondent the survey forwarding question screen] Take a look at this screen.

[If respondent did not see during the survey] If you had said that you did not include everyone, you would have seen this screen.

What are your thoughts on this screen?

Would you feel comfortable sharing this information about others in your household? [If not] Can you tell me more about that?

If you enter contact information for someone else at your address, what do you think will happen next?

Please go to the next page.

Please look at this email message. What are your thoughts about this message. What does the recipient have to do next to answer the survey?

[Show respondent the vacancy screen they saw] Take a look at this screen. You answered______. Tell me about that.

What does the word vacant mean to you in this question?

[Show respondent the vacancy screen they didn’t see] Now here is another way of asking that question. Take a look at this version. How would you answer this question?

Between these two different vacancy questions, which do you prefer, if either?

[Show vacancy help modal] Now look at this pop up window with help text and let me know your thoughts. You would see this window if you selected “Click here to find out why we are asking this question” on the page.

How helpful do this information is?

Knowledge Check

Now we would like you to complete a few more short questions before we wrap up. Please go to the next page.

Online, Paper, Or in-Person

If you were filling out this survey at home, would you prefer to complete it online, with a paper form, or talking to someone over the phone?

[IF ONLINE:] Would you prefer to do this on a computer, on your phone, on a tablet, or a different device? (If different device, please specify)

Paper Combined Race/ethnicity question

We are just about done. Please go back to your paper materials and find the piece of paper labeled Combined Race/Ethnicity Question. Pretend this question is part of a paper questionnaire. Answer that question for yourself and think aloud as you do since I can’t see how you are answering it.

After the participant has answered the question. Do you see the phrases “Enter for example” on the paper? What does that mean to you?

Okay, that’s it. Is there anything else that you’d like to share with us about the questions you have answered for us today?