2020 Census Billinugal Vietnamese, Russian, and Arabic

TO09: Project 4 Round 2 Protocol Guide – VT/RU/AR

Project 4 Round 2b Protocol – Vietnamese / Russian / Arabic

Cognitive Testing Protocol Guide

with language-specific probes

Revision History

Date	Version Number & Purpose	Author	Submitted by
8/18/2015	Eng_v0.1 Language-specific drafts	Jiyoung Son, Vietnamese language team lead Alisú Schoua-Glusberg, co-lead researcher
8/24/2015	Eng_v1.0 Draft protocol submission to Census Bureau	Mandy Sha, project manager	Mandy Sha, project manager
8/24/2015	Eng_v2.0 Revised protocol for OMB submission	Mandy Sha, project manager	Mandy Sha, project manager

Project 4 Round 2b Protocol – Vietnamese / Russian / Arabic

Participant ID: _____________________________ Interviewer ID: ______________________________

Interview language: a. Vietnamese b. Russian c. Arabic

Practice condition: Condition 1 Condition 2

Interview date: ____/____/______ (mm/dd/yyyy) Start time: ___________ AM / PM

SECTON I. INTRODUCTION AND CONSENT

Hello, my name is ___________. I work for RTI International. Thanks for agreeing to help us today. Let me start by telling you a little about what we will be doing.

• Every ten years the United States conducts a census, or count, of everyone who is living in the United States. The Census Bureau has developed the census form in multiple languages that will be mailed to people who request them across the United States. They are interested in knowing how people like you would fill it out.

• They have hired us, RTI, to conduct this research. Your participation is very important because it will help the Census Bureau determine if these forms are translated well and are culturally appropriate.

• I am going to show you a Census form, and I will ask you to review the materials and fill out the form as if you had received them in the mail and pretending I am not here.

• After you review the materials and fill out the form, I will ask you some questions about different aspects of the materials.

• When we are done, I will give you $40 as a token of our appreciation, and I will ask you to sign a receipt to document that you have received it.

PLACE THE CONSENT FORM IN FRONT OF PARTICIPANT

Before we start, I would like you to read over the document in front of you. This document explains a little bit about this interview and provides information about your rights as a participant. In addition, at the bottom of the page, we ask for your permission to have this session recorded. Please ask me any questions you have about this document. Once you have finished reading the document, please sign it.

PARTICIPANT READS AND SIGNS FORM. PROVIDE A COPY OF THE FORM TO THE PARTICIPANT

Here is a copy of this form for you to keep.

IF PARTICIPANT PROVIDES CONSENT TO HAVE THE SESSION TAPED: I will now turn on the tape recorder.

TEST TAPE RECORDER AND TURN IT ON AGAIN AFTER TEST

Now, I would like to confirm on tape that it is fine with you to record this interview.

 VERBAL CONSENT MUST BE RECORDED

Section I. End Time: _____ : _____ AM/PM

SECTION II. COGNITIVE INTERVIEW PRACTICE

For both “Condition 1 & Condition 2”:

First, I’ll ask you to fill out the Census questionnaire just as you would if you were at your home. After you complete the questionnaire, I will ask you to tell me a bit more about your answers to the Census questionnaire. The reason for this is to see how people understand each question in the questionnaire.

For example, let's say one of the questions you completed asked “How many TVs do you have? And let’s say you answered 1. After you complete the whole questionnaire, I will ask you to tell me a bit more about the answer of 1 TV.

For example, I may ask you “How did you answer this question?” I would like you to elaborate on how you decided on your answer. You could say “Well I have TV that I watch every day. I also have an old TV that doesn´t work. I assume they want to know how many working TVs I have, so I answered that I just have one TV.”

Some of the things I will ask you may sound a little strange. For example, I might ask you what a certain word means to you or how you came up with your answer.

Even though some of the things I will ask you may sound like I am giving you a test, there are no right or wrong answers. We are just interested in how the questions in the questionnaire work for you.

If “Condition 2”, continue with practice question below].

Condition 2. Start time: _____ : ________ AM/PM

Now are going to do a practice question. [IWR: Hand the practice question to R] Please try to answer it just as you would if you were at home.

1. How many times have you seen a doctor last year?

Number of times =

[IWR: Wait for R’s answer, and say] You wrote [ ] times. How did you come up with this answer?

[IWR: If R not giving enough detail, ask] Could you tell me what you’re thinking of while coming up with your answer?

PROBE IF NECESSARY:

• Which period did you think of when you responded to this question?

• What does the word “doctor” mean to you in the context of this question?

[IWR: Provide feedback to Rs whether they answered probes as intended.]

• By listening to responses in all the interviews, we find that some people understand “last year” as “the last calendar year like 2014” while others understand it as “since September/October 2014 until now”. And when thinking of “doctors”, some people included visits to the dentists and chiropractors while others do not. Your responses will help us to find out which questions are confusing and how we should improve them.

End time: _____ : ________ AM/PM

Do you have any questions before we begin?

SECTION III. COGNITIVE INTERVIEW

Here is the form. [iF INTERVIEW NOT AT R's HOME, READ] in places where the form says "this house, apartment, or mobile home," we're assuming it is where you usually live and call it home.

INTERVIEWER Instructions:

• Ask R for his or her address and write in the white space on the back page.

• Hand the form to R folded with back page white spacing showing address.

• Have R complete the form without probing

• Note any usability issues (i.e., having problems filling out the form) and confusion/difficulty (i.e., skip errors, checking wrong boxes) so that you can probe on them later

• Administer scripted probes after R completes the whole census form.

End time: _____ : ________ AM/PM

Now I have some specific questions about how you understood some of the words and phrases that were used in the questionnaire.

INTERVIEWER Instructions:

• Note any changes to the written answers which are made during probing (i.e., change content, when the change was made, R’s comments to explain the changes.

• ARABIC ONLY: Probe when the respondents did not follow the intended layout of the questionnaire, such as using write-in boxes/alignment of questions/formatting.

1. Observation (i.e., did R use a blue/black pen? Sign of confusion/difficulty? Any comments)

RU1 [Russian only]. Please look here at the top where it says 2016 Census Test. What do you think that means?

1. How much of these instructions did you read before you answered Q1?

2. Let’s re-read the first set of instructions (Before you answer Question 1….). In your own words, what is the first bullet asking you to do?

3. What do you think the second bullet instruction of the first paragraph “If no one lives or sleeps at this address, go to https://survey.census.gov/censustest or call the number on page 8” means?

4. According to this instruction, what number would you call if no one lives or sleeps at this address?

5. Let’s re-read the second set of instructions (The Census Bureau also conducts…). According to these instructions, who should have been included on Sept 1, 2015? IF NECESSARY: Can you give me some examples?

6. According to the instructions, who should have not been included on Sept 1, 2015? Can you give me some examples? IF NECESSARY: Do you think one should include kids who live in an apartment apart from the family to attend the college?

7. What does the phrase “leave these people off your form” mean here in the last bullet instruction?

8. Let’s re-read the last instruction (The Census Bureau must also include…). What do you think they mean by “If someone who has no permanent place to stay is staying here on Sept 1, 2015, count that person.”? IF NECESSARY: Can you give me an example of a person who has no permanent place to stay?

   Observation

9. Can you tell me how you arrived at your answer? Who did you include in the form?

RU2 [Russian only]. (Probe about reordered ‘house’ and ‘apartment’ terms. Did R think they had to count everyone in apartment building?] What do you think they mean here by [mobile home]? What are they referring to? And what if it said [mobile home alternative term from 2010 census]? Is that the same or different? IF DIFFERENT, How so?

Observation

10. IF R DID NOT CHECK “NO ADDITIONAL PEOPLE”: I noticed that you checked ….. in Q2. Were you thinking of people that you included in Q1 or were these different people?

RU3 [RUSSIAN ONLY] [IWR: SHOW ALTERNATIVE WORDING ON “MARK ALL THAT APPLY”] What does this instruction ask you to do? How would you mark your answer according to this instruction? How is it different from the instruction on the form? Which is better to use and why?

	Observation
	Observation

P10a. According to this question, in what situation will the Census Bureau call you? Here’s another way to write this instruction. (Show alternate wording.) Do you think the meanings of these two are the same or different? IF SAME: Which wording is easier to understand?

SHOWCARDS

A: We will only contact you if needed for official Census Bureau business. [Round 2]

B: We may call if we don’t understand an answer. [Round 1]

RU4 [RUSSIAN ONLY] What do you think they mean here by ‘equity’? And what about the term ‘mortgage’?

RU5 [RUSSIAN ONLY] What do you think they mean here by ‘rented’?

Observation

11. [POINT TO Q5] What do you think this instruction is asking you to do? Did you fill out the name in English?

12. How did you decide who Person 1 is? Did you have any difficulty deciding who to list as Person1? IF NECESSARY: Does [PERSON 1] live in this household?

RU6 [RUSSIAN ONLY] I noticed you (did / did not) enter the first letter of your patronymic. Please tell me why.

AR1 [ARABIC ONLY] What did you write in the space marked “MI” (الحرف الأول من الاسم الأوسط \ الأب the first letter of the middle name/father)? What do you think it is asking you to do here?

	Observation
	Observation

13. Do you have any babies under 1 year old in your household? IF YES: Did you have any trouble with how to record their age? Please explain what made you confused.

14. Did you see the instruction “For babies less than …..”?

15. IF R DOESN’T HAVE BABIES UNDER 1 YEAR OLD: The instruction in Q 7 says: “For babies less than 1 year old, do not write the age in months. Write 0 as the age.” If a person has a 4-month old baby girl, what age should she write here?

Observation

16. Can you tell me what this question was asking you?

17. How did you choose that answer? [FOR ARABIC: IF MENA NOT CHOSEN, does this category applies to you? Who do you think should choose this category?]

18. IF R DID NOT MARK ANY RACES: You did not make any marks here. Why did you choose to do that?

19. If R MARKED A RACE BUT DID NOT WRITE IN: You did not write in anything. Why did you choose to do that?

20. Based on the instruction here [point to the instruction], how many categories do you think it’s ok to check here?

21. IF R HAS CHOSEN ONLY ONE CATEGORY: What do you think a person should do if they are half Vietnamese and half Italian?

22. When you first read the question, did you see the Note about reporting more than one group?

23. IF R HAS REPORTED MORE THAN ONE GROUP: How would you decide which group to mark or write if you need to choose only one group?

VT 1. [VIETNAMESE ONLY] Let’s look at this option, “Người Châu Mỹ La-tinh nói tiếng Tây Ban Nha (Spanish speaking Latin American continent person).” What do you think it means?

Now I am going to show you another phrase. [IWR: PRESENT THE SHOWCARD #1] Do you think the two mean the same or different? [IF SAME] Which one is easier to understand?

VT SHOWCARD #1

A: Người Châu Mỹ La-tinh nói tiếng Tây Ban Nha (Spanish speaking Latin American continent person)

B: Hispanic, Latino, hoặc Tây Ban Nha (Hispanic, Latino, or Spanish)

Observation (i.e., did R mark both “no” and “yes”?)

24. Can you tell me why you chose …..?

25. FOR ANY ANSWER WITH A MARKED CATEGORY: What place were you thinking of? How long (do you/does [NAME]) stay there?

[ALL LANGUAGES] What does “for a military assignment” mean in this question?

[ALL LANGUAGES] What does “with a parent or other relative” mean in this question? When someone lives somewhere else sometimes with a parent or other relative, does that someone have to be a child, an adult or both? Can you give me an example?

VT3 [VIETNAMESE ONLY] Please look at the option, “Nhà nghỉ mát hoặc nhà ở khác (Vacation home or another home)—translation for “At a seasonal or second residence”). What does it mean in this question?

Now, I’m going to show you another phrase. What does this phrase (B) mean to you? Do you think the two wordings are the same or different? How so?

VT SHOWCARD #2

A: Nhà nghỉ mát hoặc nhà ở khác (Vacation home or another home)

B: Tại nơi ở theo thời vụ, hoặc nơi ở thứ hai (residence for certain seasons/specific period, or second residence)

[All Languages]] Did you notice this instruction, “Skip to Person 2 if more people live here” while you were filling out the form? What do you think it is asking you to do?

[Arabic only] Which sounds more natural?

• Option 1 (on the form):

لا ← توقف. انتقل إلى الشخص الثاني إذا كان هناك أشخاص أكثر.

(No → Stop. Move to the second person (2) if there are more people.)

• Option 2 (Show Card):

لا ← توقف. انتقل إلى الشخص الثاني إذا كان هذا الشخص يعيش هنا.

No → Stop. Move to the second person (2) if this person lives here.

Observation (i.e., Did R understand the skip instruction? Did R provide the address appropriately?)

[ARABIC AND RUSSIAN] Check for address completeness.

• (Arabic) Probe about رقم البيت

Home number (For example: 5007)

26. Okay, now let's look at Question 10. What did you think this question was asking you to do?

27. What do you think “Rural Route Address (if there is no street address)” means here?

28. What do you think the phrase “if there is no street address” means here? IF NEEDED: Did this affect how you answered this question?

VT4. [VIETNAMESE ONLY] [IF R FILLED OUT THE ADDRESS, ASK] Was it easy or difficult for you to fill out “Mã bưu chính (Zip Code)?” [IF R DID NOT FILL OUT THE ADDRES, ASK] If someone has to write the address here, what should he/she write in here [point Mã bưu chính (Zip Code) (Postal Code)]?

29. [POINT TO COUNTY FIELDS] What is this field asking you to write? Did you notice these English words in parenthesis when you answered this question? Do these English words help you when you filled out the form? Why or why not?

VT 5. [VIETNAMESE ONLY] I would like to show you another way to express this. Is the second wording (B) different from the first (A)? How so? If you are asked to write your address on a form, which one would you prefer to see on the form? Can you think of any better way to put it so people who fill out this form can write in their County easier?

VT SHOWCARD #3

A: Quận/Vùng/Giáo xứ (County, Parish, Borough)

B: Quận, Khu Phố, Thị xã (County, Parish, Borough)

30. [POINT TO THE “NOTE”] Did you notice the note when you filled out the form? What is this telling you to do?

31. IF R SKIPPED IN ERROR: You didn’t mark anything here. Why did you choose to do that?

32. IF R WROTE IN Q10 IN ERROR: How did you choose to write this address here?

33. IF R WROTE IN Q10 CORRECTLY: How did you choose to write this address here?

Observation (i.e., Did R understand the skip instruction?)

34. Can you tell me how you chose that answer?

35. IF R DID NOT MARK A RESPONSE IN ERROR: You didn't mark anything here. Could you tell me what you were thinking?

36. IF R CHOSE “The address or location you listed in Question 10”, but Q10 is blank. You chose “The address or location you listed in Question 11”, but you did not provide any address at Q10. Could you tell me what you were thinking?

37. IF R CHOSE “Both places equally”, but Q10 is blank. You chose “Both places equally”, but you did not provide any address at Q10. Could you tell me what you were thinking?

38. IF R CHOSE “SOME OTHER PLACE”: Could you tell me more about this other place and the time that (you/PERSON 1 NAME) spend(s) there?

VT6. [VIETNAMESE ONLY] What do you think “thường xuyên” (most frequently/most of the time)” means here?

Observation (i.e., Did R understand the skip instruction?)

39. Could you tell me why you chose that answer?

40. IF R DID NOT MARK A RESPONSE: You didn't mark anything in Question 12. Could you tell me what you were thinking?

41. IF R CHOSE “The address or location you listed in Question 10”, but Q10 is blank: You chose “The address or location you listed in Question 10”, but you did not provide any address at Q10. Could you tell me what you were thinking?

42. IF R’ CHOSE “SOME OTHER PLACE”: Can you tell me more about this other place and the time that (you/NAME) spend(s) there?

    Observation (i.e., Did R understand the skip instruction?)

43. Did you see this instruction when you fill out the form by yourself? What do you think this instruction is asking you to do?

	Observation (i.e., what did R write?)
	Observation (i.e., what did R write?)
	Observation (i.e., what did R write?)
	Observation (i.e., what did R write?)
	Observation (i.e., what did R write?)

AR2 [ARABIC ONLY]. How did you decide who to list as Person 2? [point to Person 2 label on top of the page] What did you think this meant?

Person	Relationship to Person 1	PROBE: How easy or difficult was it to find a category that fit (NAME)? IF NECESSARY: Did you look at or consider any of the other categories? Was there something you were looking for in the list that you could not find?
Person2	Observation
Person3	Observation
Person4	Observation
Person5	Observation
Person6	Observation

49. IF R IS NOT PERSON 1: Were you thinking about yourself or [Person 1 Name] when you marked the relationship for (NAME)?

RU7 [RUSSIAN ONLY] What do you think is meant by ‘child on patronate / foster’?

AR3 [ARABIC ONLY] What do you think they mean by:

كفالة الطفل من خلال البرنامج الحكومي Foster

(A child being ensured of care through the government program Foster)

What about:

طفل تلقى رعاية من خلال البرنامج الحكومي Foster(A child that received care because of the government program Foster)

Do they mean the same or different?

[VIETNAMESE ONLY VIGNETTE INTRO] Now, I have a few situations that I would like to discuss with you, to see how someone like that might answer to some of the questions in this form. Here is a copy of what I am reading in case you would like to follow along. [IWR: HAND R COPY OF VIGNETTE TEXT ONE AT A TIME]

VT7. [VIETNAMESE ONLY—Vignette 1] Huyen lives with her uncle because the uncle’s house is close to her work. If the uncle, who is the owner of the house fills out the form, which category best describes Huyen’s relationship to the uncle and why? Was it easy or difficult to find an answer for Huyen?

[IF THE ANSWER WAS NOT “Họ hàng khác (Ví dụ: như cậu, cháu, v.v….)” (Other relative (Ex: uncle, nephew, …)), ASK] Would you say that Huyen is the uncle’s “Other relative”? How so?

VT 8. [VIETNAMESE ONLY—Vignette 2] Two men, Van and Minh, lived together and were married as soon as it became legal where they live. Which category best describes Van’s relationship to Minh and why? Was it easy or difficult to find an answer for Van?

[IF THE ANSWER WAS NOT “Người phối ngẫu cùng giới là chồng/vợ” (same sex spouse who is husband/wife), ASK] How is “Người phối ngẫu cùng giới là chồng/vợ” (same sex spouse who is huband/wife) different from “Người phối ngẫu khác giới là chồng/vợ” (opposite sex spouse who is husband/wife)?

VT9. [VIETNAMESE ONLY] We are trying to find the best way to describe this relationship in Vietnamese. Please take a look at this page [IWR: HAND VT-SHOWCARD #4 TO R]. Which one sounds most natural in Vietnamese to you?

VT SHOWCARD #4

A: Người phối ngẫu khác giới là chồng/vợ (opposite sex spouse who is husband/wife)

B: Người phối ngẫu khác giới (spouse of the opposite-sex)

C: Chồng/vợ khác giới (husband/wife of the opposite-sex)

VT10. [VIETNAMESE ONLY] How about this? [IWR: PRESENT VT-SHOWCARD #5] Please also take a look and tell me which one sounds most natural in Vietnamese to you?

VT SHOWCARD #5

A: Người phối ngẫu cùng giới là chồng/vợ (same sex spouse who is husband/wife)

B: Người phối ngẫu cùng giới (spouse of the same sex)

C: Chồng/vợ cùng giới (husband/wife of the same sex)

VT11. [VIETNAMESE ONLY] What does ““Người sống chung không đăng ký kết hôn” (a person living together without registering for marriage [unmarried partner]) mean here? Does it sound natural to you? Could you think of any other way to describe this term in Vietnamese? What does “Người cùng giới sống chung không đăng ký kết hôn” mean to you here?

24. Observation

P50. What country (were you/was [NAME]) born in? COLLECT DATA FOR ALL PERSONS LISTED IN THE QUESTIONNAIRE, P1 –P6.

P49_Person 1: _______________________

P49_Person 2: _______________________

P49_Person 3: _______________________

P49_Person 4: _______________________

P49_Person 5: _______________________

P49_Person 6: _______________________

Tenure Vignette.

Now, I have a situation I would like to run past you, to see how you think someone in that situation might answer one of the survey questions: 

Mrs. Smith is responding to the census questionnaire for her household. She lives with her son, who pays the mortgage, but she doesn't pay anything. 

P51. How should she respond?  How did you come up with that answer?

P52. What does “occupied without payment of rent” mean to you in this question? Can you give me an example of when someone should pick this option?

P53. Now that you have filled out the census form, what do you think is the most important use of the Census data?

P54. How do you think the census data may benefit your community? And do you see any benefit of the census data at the personal level?

SECTION IV. FINAL DEBRIEFING

Thanks, we’re almost finished. I have a few more questions for you.

Language fluency	Very Well	Well	Not well	Not at all
DB1. How well do you speak English?
DB2. How well do you read English?
DB3. How well do you write in English?
DB4. How well do you understand movies and videos in English?

Language use	1.Almost never	2.Some-times	3.Often	4.Almost always
DA1. How often do you think in English? Would you say…
DA2. How often do you speak English at home? Would you say…
DA3. How often do you speak English with your friends? Would you say…

FOR ALL INTERVIEWS: (Skip if interview longer than 60 minutes)

1. Aside from the things we’ve already talked about, were there any questions or instructions that you found confusing or difficult to answer?

2. Is there anything else you would like to tell me that you haven’t already?

Thank you very much for your help today. PAY R AND HAVE THEM SIGN RECEIPT. TURN OFF RECORDER.

End Time: ___________ AM / PM

[INTERVIEWER: Answer the following questions right after the interview]

1. How comfortable do you think the respondent was to answer the probe questions in general?

Not comfortable at all		Somewhat comfortable		Very comfortable
1	2	3	4	5

2. How well do you think the respondent understood the purpose of the cognitive testing?

Poor understanding		Good understanding		Excellent understanding
1	2	3	4	5

The next questions are about how the respondent answered the probes (e.g. what does x mean to you in this question? How did you come up with this answer?)

3. How frequently did you have to ask the respondent to provide more detail about their answers (eg., asking follow-up probes for clarification)?

Never	Seldom	About half the time	Usually	Always
1	2	3	4	5

4. How would you rate the respondent’s performance in answering the probe questions as intended at different stages of the cognitive interview?

4-1. Beginning (Section IV-A)

Poor job				Excellent job
1	2	3	4	5

4-2. In the middle (Section IV-B)

Poor job				Excellent job
1	2	3	4	5

4-3. Toward the end (Section IV-C)

Poor job				Excellent job
1	2	3	4	5

OMB Control Number: 0607-0725; Expiration: 8/31/16