Form 1 Protocol

Pre-Testing of GQ resident emails and online instrument

RESPONDENT ID #: _________________________ DATE: ____ / ____ / _____

INTERVIEWER’S NAME: _________________________________________________

START TIME: ______: ______ AM / PM UHE VERSION: _____________

LOCATION: _____________

Introduction

Hello, my name is ________________, and I work for the Census Bureau. Thank you for agreeing to help us today. Let me start by telling you a little about what we will be doing today.

Every year, the Census Bureau conducts a number of surveys. Today we are going to look at some emails and the online questionnaire that you would answer online if you were completing the census.

Because these are example emails, they may not look quite the same they would if you received them in your inbox. We will go through different emails one by one. After you’ve finished with one email, I will have you look at the next.

As we go through these emails, please treat each one as you would if it came to your inbox and as if I were not there. I’d like you to say out loud what you are thinking as you look at the emails.

At the end, I’m also going to ask you some questions about the emails and about your reactions to them. There are no right or wrong answers. I really want to hear your opinions and reactions to these emails, so don’t hesitate to speak up whenever something is unclear. Please feel free to be honest.

[IF OBSERVER PRESENT]: I want to let you know that this interview is being observed by other Census Bureau staff. They are observing the interview because they are also interested in how the mailings work for people like you.

Do you have any questions about what we’ll be doing?

Informed Consent

• CONSENT FORM.

[Participant was emailed consent form]. We emailed you information about the session today and your rights as a participant. That document explains a little bit about this interview and provides information about your rights as a participant, that your participation is optional, all information you provide is confidential, and we won’t use your name in any report. It also asks for your permission to have this session audio recorded and for your screen to be recorded so that I can concentrate on what you are saying rather than having to take extensive notes. Please ask me any questions you have about this document.

I need to capture your consent orally. Only those of us connected with the project will review the recording and it will be used solely for research purposes. Your name will not be associated with the recording or any of the other data collected during the session. Do you give your consent? Please repeat this statement “I give consent to be audio recorded and my screen to be recorded.”

• [IF RESPONDENT PROVIDES CONSENT TO HAVE THE SESSION RECORDED]: I will now turn on the recorder.

• [TURN ON RECORDER]

Think Aloud

I’d like you to answer the questions as you would if you were at home without me there, but with one big difference. What I would like you to do is tell me what you are thinking and feeling as you come up with your answers. 

Let’s do a practice question before we start: Please think aloud as you answer the question, how many windows are in your home?

[GIVE FEEDBACK ON THE THINK ALOUD THEY DID]

Camera

So, just like we practiced in the tech check – I am going to need you to go ahead and turn your camera on by clicking on the camera icon.

I’m going to put a link in the chat and I’d like for you to select that and then share your screen.

[If able to see P] Okay great I can see you. Now we can get started with the task. For this task you are going to look at some of our census notifications. We plan to send them by email. These emails reference the real census date of 2030. You’ll have to pretend that it is 2030 and you are living in your current residence.

[If unable to see P will need to troubleshoot it]

Initial email: Okay, let’s get started and you can go to the next page.

Please pretend that you received this email with this subject line, what would you do? Remember to think out loud as you do.

[ ] [If said would open it.] Okay, you can go to the next page. And this is the email itself. Remember to keep talking.

[ ] [If said would not open it.] Thank you for your honesty. Let’s pretend that you did open it anyway. Please go to the next page. And this is the email itself. Remember to keep talking.

[If participant reads aloud.] You do not have to read aloud unless that is what you would do at home.

OBSERVE (DO NOT READ TO PARTICIPANT):

1. What does P look at (or comment on) first?

___

2. Does P comment on “required by law”?

___ Yes ___ No ___ Not sure

3. Does P comment on why their response matters?

___ Yes ___ No ___ Not sure

4. Does P comment on 5-minute survey length?

___ Yes ___ No ___ Not sure

5. Does P spontaneously mention wanting to know when it’s due?

___ Yes ___ No ___ Not sure

6. Does P spontaneously comment about whether it’s confidential?

___ Yes ___ No ___ Not sure

7. Does P comment on the lack of URL or link?

___ Yes ___ No ___ Not sure

8. Does P comment on email subject line?

___ Yes ___ No ___ Not sure

9. Does P comment on Rob Santos?

___ Yes ___ No ___ Not sure

10. Does P comment on email as a notification mode?

___ Yes ___ No ___ Not sure

Note any questions P asked/comments P made about this email:

Other notes:

Email 2:

Please go to the next page.

Please pretend that you received this email with this subject line, what would you do? Remember to think out loud as you do.

[ ] [If said would open it.] Okay, you can go to the next page. And this is the email itself. Remember to keep talking.

[ ] [If said would not open it.] Thank you for your honesty. Let’s pretend that you did open it anyway. Please go to the next page. And this is the email itself. Remember to keep talking.

[If participant reads aloud.] You do not have to read aloud unless that is what you would do at home.

OBSERVE (DO NOT READ TO PARTICIPANT):

1. What does P look at (or comment on) first?

___

2. Does P comment on the quantity of emails?

___ Yes ___ No ___ Not sure

3. Does P comment on the email subject line?

___ Yes ___ No ___ Not sure

4. Does P comment on the URL?

___ Yes ___ No ___ Not sure

Note any questions P asked/comments P made about this email:

Other notes:

[If P does not provide enough information during think aloud] Do you have any reactions or any immediate thoughts about what you just read? 

Thank you.

Email 3:

Please pretend that you received this email with this subject line, what would you do? Remember to think out loud as you do.

[ ] [If said would open it.] Okay, you can go to the next page. And this is the email itself. Remember to keep talking.

[ ] [If said would not open it.] Thank you for your honesty. Let’s pretend that you did open it anyway. Please go to the next page. And this is the email itself. Remember to keep talking.

1. What does P look at (or comment on) first?

___

2. Does P comment on the quantity of emails?

___ Yes ___ No ___ Not sure

3. Does P comment on the email subject line?

___ Yes ___ No ___ Not sure

4. Does P ask whether they would get this email even if they completed their form?

___ Yes ___ No ___ Not sure

5. Does P comment on the URL?

___ Yes ___ No ___ Not sure

Email 4:

Please pretend that you received this email with this subject line, what would you do? Remember to think out loud as you do.

[ ] [If said would open it.] Okay, you can go to the next page. And this is the email itself. Remember to keep talking.

[ ] [If said would not open it.] Thank you for your honesty. Let’s pretend that you did open it anyway. Please go to the next page. And this is the email itself. Remember to keep talking.

1. What does P look at (or comment on) first?

___

2. Does P comment on the quantity of emails?

___ Yes ___ No ___ Not sure

3. Does P comment on the email subject line?

___ Yes ___ No ___ Not sure

4. Does P ask whether they would get this email even if they completed their form?

___ Yes ___ No ___ Not sure

5. Does P comment on the URL?

___ Yes ___ No ___ Not sure

Email 5:

Please pretend that you received this email with this subject line, what would you do? Remember to think out loud as you do.

[ ] [If said would open it.] Okay, you can go to the next page. And this is the email itself. Remember to keep talking.

[ ] [If said would not open it.] Thank you for your honesty. Let’s pretend that you did open it anyway. Please go to the next page. And this is the email itself. Remember to keep talking.

1. What does P look at (or comment on) first?

___

2. Does P comment on the quantity of emails?

___ Yes ___ No ___ Not sure

3. Does P comment on the email subject line?

___ Yes ___ No ___ Not sure

4. Does P ask whether they would get this email even if they completed their form?

___ Yes ___ No ___ Not sure

5. Does P comment on the URL?

___ Yes ___ No ___ Not sure

Debriefing Probes

Thank you. Now I have a few questions I’d like to ask you about the emails.

Only ask probe if not mentioned during think aloud.

1. Do you personally use email?

   1. What do you use it for?

2. How often do you check your email?

3. Would you open these emails?

4. Would you be more or less likely to open the email if the sender was from your organization or from the U.S. Census Bureau?

5. How many emails reminders would you want?

6. [SHOW FIRST EMAIL] This is the first email. What is the purpose of this email?

   1. (If don’t mention) There is no web survey link or ID in this email. What are your thoughts about that?

   2. What are your thoughts about who the email is from – Robert Santos? [Point to name].

      1. Does having that name on the email make you more or less likely to respond?

      2. Would having an actual signature instead of the typed name matter to you?

7. [SHOW 2nd EMAIL] This is the next email. What is the purpose of this email?

   1. What do you think about the link to the survey? Would you have any concerns about clicking this link?

8. [SHOW 3rd EMAIL] This is the next email. What is the purpose of this email?

9. [SHOW 4th EMAIL] This is the next email. What is the purpose of this email?

10. [SHOW 5th EMAIL] This is the next email. What is the purpose of this email?

11. Of the emails you saw today, which ones did you feel were the easiest to understand?

12. Were there any emails which you found confusing? If so, which ones and what was confusing?

13. What do you think the main point of these emails? [Do they understand that they have to complete the census?]

14. Any other comments on these emails?

15. In your opinion, do you think email is the best way for you to be notified about the online census questionnaire?

    1. What would be better?

16. [If any observations marked as yes above] I noticed you/you commented on…

Well, that’s all the questions I have about the emails. Thank you so much for your feedback.

Let’s pretend that you decided to click the link in the emails. So, please click the link and it will take you to the survey.

Go to the next screen to see the URL – you can select that now.

For the remainder of the session, you’ll answer the online census questionnaire that you read about in the emails. Then we will have you answer a few questions about your experience. I want you to respond to it naturally but just like you have been doing so far, remember to think aloud as you do it. If you fall silent, I may remind you to think aloud by saying “keep talking” or “what are you thinking?” On the next page is a link to another survey. Please select the link and answer the survey while thinking aloud.

URL for Qualtrics Decennial Questionnaire:

Make note if they had any difficulty with verifying their address.

Make note of any difficulty understanding the UHE version.

UHE Question for debriefing.

OBSERVE (DO NOT READ TO PARTICIPANT):

1. What version address did they receive?

OBSERVE (DO NOT READ TO PARTICIPANT):

1. Did P answer correctly?

___

2. Did P comment on the question?

___ Yes ___ No ___ Not sure

What was the comment?

2. UHE Question

OBSERVE (DO NOT READ TO PARTICIPANT):

1. Did P select another place?

___ Yes ___ No ___ Not sure

2. Did P comment on the question?

___ Yes ___ No ___ Not sure

What was the comment?

3. What version of the UHE did they receive?

OBSERVE (DO NOT READ TO PARTICIPANT):

1. Did P answer correctly?

___

2. Did P comment on the question?

___ Yes ___ No ___ Not sure

What was the comment?

Now I’m going to have you answer some satisfaction questions about the survey you just completed. I’m going to put another link in the chat.

URL for Satisfaction/Debriefing/Quizlette/:

URL

[HAVE RESPONDENT COMPLETE THE SATISFACTION QUESTIONS IN QUALTRICS]

Debriefing

[HAVE RESPONDENT REVIEW THE SCREENS OF NOTE IN QUALTRICS]

Now I just have a few more questions for you about the survey.

ADDRESS

We have three different versions of this screen.

You saw this one (tell them the first second or third).

What does this question mean to you?

Now, let’s look at the other versions and get your reactions.

What does this question mean to you?

What would your answer be to that question?

Now go to the other one…

What does this question mean to you?

What would your answer be to that question?

Which of the three to you prefer if any?

Please tell me more about why you prefer that one?

Are there any versions you do not like as much?

Tell me more about why you don’t like that version.

UHE

You saw this screen and answered FILL Yes or No.

What does this question mean to you?

If yes: Tell us more about how often you go to that other address.

If no – tell us more about why you answered no.

UHE most of the time

We have two different versions of this screen.

You saw this one (tell them the first or second).

What does this question mean to you?

Now, let’s look at the other version.

What does this question mean to you?

What would your answer be to that question?

Which of these questions do you think is the easiest to understand?

Knowledge check

We’re almost done. Now we would like you to complete a few more short questions before we wrap up and I’m going to check to see if any of our observers have questions.

[HAVE THE RESPONDENT COMPLETE THE KNOWLEDGE CHECK QUESTIONS IN QUALTRICS]

Voucher

We’re almost done. Now we would like you to complete a few more short questions before we wrap up.

We will send out your incentive money via Priority Mail by the end of the week, so please enter the address where you can receive mail. We will email you the tracking number.

[HAVE THE RESPONDENT COMPLETE THE Voucher IN QUALTRICS. Make sure the mailing address is good.]

Thank you very much for participating in today’s session.