2020 Chinese Korean Bilingual Forms

TO09: Project 4 Round 2 Protocol Guide – CH/KO

Project 4 Round 2 Protocol – Chinese and Korean

(Flip style)

Revision History

Date	Version Number & Purpose	Author	Submitted by
8/5/2015	V1.0	Hyunjoo Park, co-lead Researcher & Yuling Pan, Sociolinguist	Mandy Sha, project manager
8/7/2015	V2.0	Hyunjoo Park, co-lead Researcher & Yuling Pan, Sociolinguist	Mandy Sha, project manager

Cognitive Testing Protocol Guide

Participant ID: _____________________________ Interviewer ID: ______________________________

Interview language: 1. Chinese 2. Korean

Practice condition: Condition 1 Condition 2

Interview date: ____/____/______ (mm/dd/yyyy) Start time: ___________ AM / PM

SECTON I. INTRODUCTION AND CONSENT

Hello, my name is ___________. I work for RTI International. Thanks for agreeing to help us today. Let me start by telling you a little about what we will be doing.

• Every ten years the United States conducts a census, or count, of everyone who is living in the United States. The Census Bureau has developed the census form in multiple languages that will be mailed to people who request them across the United States. They are interested in knowing how people like you would fill it out.

• They have hired us, RTI, to conduct this research. Your participation is very important because it will help the Census Bureau determine if these forms are translated well and are culturally appropriate.

• I am going to show you some materials and a Census form, and I will ask you to review the materials and fill out the form as if you had received them in the mail and pretending I am not here.

• After you review the materials and fill out the form, I will ask you some questions about different aspects of the materials.

• When we are done, I will give you $40 as a token of our appreciation, and I will ask you to sign a receipt to document that you have received it.

PLACE THE CONSENT FORM IN FRONT OF PARTICIPANT

Before we start, I would like you to read over the document in front of you. This document explains a little bit about this interview and provides information about your rights as a participant. In addition, the back page asks for your permission to have this session recorded. Please ask me any questions you have about this document. Once you have finished reading the document, please sign it.

IWR: TURN ON YOUR LAPTOP/TABLET AND LAUNCH THE POWER POINT WHILE R IS READING

PARTICIPANT READS AND SIGNS FORM. PROVIDE A COPY OF THE FORM TO THE PARTICIPANT

Here is a copy of this form for you to keep.

IF PARTICIPANT PROVIDES CONSENT TO HAVE THE SESSION TAPED: I will now turn on the tape recorder.

TEST TAPE RECORDER AND TURN IT ON AGAIN AFTER TEST

Now, I would like to confirm on tape that it is fine with you to record this interview.

 VERBAL CONSENT MUST BE RECORDED

Section I. End Time: _____ : _____ AM/PM

SECTION II. COGNITIVE INTERVIEW PRACTICE

For both “Condition 1 & Condition 2”:

Now, we will do something a little different. First, I’ll ask you to fill out the Census questionnaire just as you would if you were at your home. After you complete the questionnaire, I will ask you to tell me a bit more about your answers to the Census questionnaire. The reason for this is to see how people understand each question in the questionnaire.

For example, let's say one of the questions you completed asked “How many TVs do you have? And let’s say you answered 1. After you complete the whole questionnaire, I will ask you to tell me a bit more about the answer of 1 TV.

For example, I may ask you “How did you answer this question?” I would like you to elaborate on how you decided on your answer. You could say “Well I have TV that I watch every day. I also have an old TV that doesn´t work. I assume they want to know how many working TVs I have, so I answered that I just have one TV.”

Some of the things I will ask you may sound a little strange. For example, I might ask you what a certain word means to you or how you came up with your answer.

Even though some of the things I will ask you may sound like I am giving you a test, there are no right or wrong answers. We are just interested in how the questions in the questionnaire work for you.

If “Condition 2”, continue with practice question below].

Condition 2. Start time: _____ : ________ AM/PM

Now are going to do a practice question. [IWR: Hand the practice question to R] Please try to answer it just as you would if you were at home.

1. How many times have you seen a doctor last year?

Number of times =

[IWR: Wait for R’s answer, and say] You wrote [ ] times. How did you come up with this answer?

[IWR: If R not giving enough detail, ask] Could you tell me what you’re thinking of while coming up with your answer?

PROBE IF NECESSARY:

• Which period did you think of when you responded to this question?

• What does the word “doctor” mean to you in the context of this question?

[IWR: Provide feedback to Rs whether they answered probes as intended.]

• By listening to responses in all the interviews, we find that some people understand “last year” as “the last calendar year like 2014” while others understand it as “since September/October 2014 until now”. And when thinking of “doctors”, some people included visits to the dentists and chiropractors while others do not. Your responses will help us to find out which questions are confusing and how we should improve them.

End time: _____ : ________ AM/PM

Do you have any questions before we begin?

SECTION III. COGNITIVE INTERVIEW

Here is the form. [iF INTERVIEW NOT AT R's HOME, READ] in places where the form says "this house, apartment, or mobile home," we're assuming it is where you usually live and call it home.

INTERVIEWER Instructions:

• Hand the form to R facing English on front

• Have R complete the form without probing

• Note any usability issues (i.e., having problems filling out the form) and confusion/difficulty (i.e., skip errors, checking wrong boxes) so that you can probe on them later

• Administer scripted probes after R completes the whole census form.

1. ANY PROBLEMS FINDING THE NON-ENGLISH FORM?

2. LANGUAGE USED FILLING OUT THE FORM?

3. HOW R HANDLED THE FORM? (i.e., Does R flip over to non-English form? Does R start filling out English form? Does R make reference to English quex when filling out non-English form?)

4. ANY COMMENT ON THE BLANK SPACE AT TOP OF PAGE 1?

IWR: Ask follow-up probes on any observed issues where the issues were probed

• I noticed when you were filling out the questionnaire, you ….., what were you thinking at that point?

• I saw you ….. , how did you decide to do that?

Now I have some specific questions about how you understood some of the words and phrases that were used in the questionnaire.

INTERVIEWER Instructions:

Note any changes to the written answers which are made during probing (i.e., change content, when the change was made, R’s comments to explain the changes.

1. Observation (i.e., did R use a blue/black pen? Sign of confusion/difficulty? Any comments)

1. How much of these instructions did you read before you answered Q1?

2. Let’s re-read the first set of instructions (Before you answer Question 1….). In your own words, what is the first bullet asking you to do?

3. What do you think the second bullet instruction of the first paragraph “If no one lives or sleeps at this address, go to https://survey.census.gov/censustest or call the number on page 8” means?

4. According to this instruction, what number would you call if no one lives or sleeps at this address?

5. Let’s re-read the second set of instructions (The Census Bureau also conducts…). According to these instructions, who should have been included on Sept 1, 2015? IF NECESSARY: Can you give me some examples?

6. According to the instructions, who should have not been included on Sept 1, 2015? Can you give me some examples? IF NECESSARY: Do you think one should include kids who live in an apartment apart from the family to attend the college?

7. What does the phrase “leave these people off your form” mean here in the last bullet instruction?

8. Let’s re-read the last instruction. What do you think they mean by “If someone who has no permanent place to stay is staying here on Sept 1, 2015, count that person.”? IF NECESSARY: Can you give me an example of a person who has no permanent place to stay?

   Observation

9. Can you tell me how you arrived at your answer? Who did you include in the form?

KR1 [KOREAN ONLY] I will show you two similar wordings. [IWR: Give R time to read Korean alternative wording #2] Do you think the meanings of these two are same or different? IF SAME: Which wording is easier to understand?

Observation

10. IF R DID NOT CHECK “NO ADDITIONAL PEOPLE”: I noticed that you checked ….. in Q2. Were you thinking of people that you included in Q1 or were these different people?

CH1 [CHINESE ONLY] [IWR: SHOW ALTERNATIVE WORDING ON “MARK ALL THAT APPLY”] What does this instruction ask you to do? How would you mark your answer according to this instruction?

	Observation
	Observation

P10a. According to this question, in what situation will the Census Bureau call you? Here’s another way to write this instruction. (Show alternate wording.) Do you think the meanings of these two are the same or different? IF SAME: Which wording is easier to understand?

Observation

11. [POINT TO Q5] What do you think this instruction is asking you to do?

12. How did you decide who Person 1 is? Did you have any difficulty deciding who to list as Person1? IF NECESSARY: Does [PERSON 1] live in this household?

    	Observation
    	Observation

13. Do you have any babies under 1 year old in your household? IF YES: Did you have any trouble with how to record their age? Please explain what made you confused.

14. Did you see the instruction “For babies less than …..”?

15. IF R DOESN’T HAVE BABIES UNDER 1 YEAR OLD: The instruction in Q 7 says: “For babies less than 1 year old, do not write the age in months. Write 0 as the age.” If a person has a 4-month old baby girl, what age should she write here?

KR1 [KOREAN ONLY] We want to know what the most natural way to ask someone’s age and date of birth is. I will show you two similar wordings [IWR: Give R time to read Korean alternative wording #3] Which wording sounds more natural to you?

KR2. [KOREAN ONLY] We would also want to know which age counting unit looks more appropriate to appear on the official document such as the census form. If you have to choose, which one would you choose between 세 [SE] and 살 [SAL]?

Observation

16. Can you tell me what this question was asking you?

17. How did you choose that answer?

18. IF R DID NOT MARK ANY RACES: You did not make any marks here. Why did you choose to do that?

19. If R MARKED A RACE BUT DID NOT WRITE IN: You did not write in anything. Why did you choose to do that?

20. How many categories do you think it’s ok to check here?

21. IF R HAS CHOSEN ONLY ONE CATEGORY: What do you think a person should do if they are half Chinese and half Italian?

22. When you first read the question, did you see the Note about reporting more than one group?

23. IF R HAS REPORTED MORE THAN ONE GROUP: How would you decide which group to mark or write if you need to choose only one group?

    Observation (i.e., did R mark both “no” and “yes”?)

24. Can you tell me why you chose …..?

25. FOR ANY ANSWER WITH A MARKED CATEGORY: What place were you thinking of? How long (do you/does [NAME]) stay there?

    Observation (i.e., Did R understand the skip instruction? Did R provide the address appropriately?)

26. Okay, now let's look at Question 10. What did you think this question was asking you to do?

27. What do you think “Rural Route Address” means here?

28. What do you think the phrase “if there is no street address” means here? IF NEEDED: Did this affect how you answered this question?

29. [POINT TO EXAMPLES AFTER COUNTY FIELDS] Did you notice these English words when you answered this question? Do these examples help you when you filled out the form? Why or why not?

30. [POINT TO THE “NOTE”] Did you notice the note when you filled out the form? What is this telling you to do?

31. IF R SKIPPED IN ERROR: You didn’t mark anything here. Why did you choose to do that?

32. IF R WROTE IN Q10 IN ERROR: How did you choose to write this address here?

33. IF R WROTE IN Q10 CORRECTLY: How did you choose to write this address here?

Observation (i.e., Did R understand the skip instruction?)

34. Can you tell me how you chose that answer?

35. IF R DID NOT MARK A RESPONSE IN ERROR: You didn't mark anything here. Could you tell me what you were thinking?

36. IF R CHOSE “The address or location you listed in Question 10”, but Q10 is blank. You chose “The address or location you listed in Question 11”, but you did not provide any address at Q10. Could you tell me what you were thinking?

37. IF R CHOSE “Both places equally”, but Q10 is blank. You chose “Both places equally”, but you did not provide any address at Q10. Could you tell me what you were thinking?

38. IF R CHOSE “SOME OTHER PLACE”: Could you tell me more about this other place and the time that (you/PERSON 1 NAME) spend(s) there?

CH3 [CHINESE ONLY] Can you tell me what the phrase “最常住” (most frequently) means in this question?

Observation (i.e., Did R understand the skip instruction?)

39. Could you tell me why you chose that answer?

40. IF R DID NOT MARK A RESPONSE: You didn't mark anything in Question 12. Could you tell me what you were thinking?

41. IF R CHOSE “The address or location you listed in Question 10”, but Q10 is blank: You chose “The address or location you listed in Question 10”, but you did not provide any address at Q10. Could you tell me what you were thinking?

42. IF R’ CHOSE “SOME OTHER PLACE”: Can you tell me more about this other place and the time that (you/NAME) spend(s) there?

    Observation (i.e., Did R understand the skip instruction?)

43. Did you see this instruction when you fill out the form by yourself? What do you think this instruction is asking you to do?

	Observation (i.e., what did R write?)
	Observation (i.e., what did R write?)
	Observation (i.e., what did R write?)
	Observation (i.e., what did R write?)
	Observation (i.e., what did R write?)

Person	Relationship to Person 1	PROBE: How easy or difficult was it to find a category that fit (NAME)? IF NECESSARY: Did you look at or consider any of the other categories? Was there something you were looking for in the list that you could not find?
Person2	Observation
Person3	Observation
Person4	Observation
Person5	Observation
Person6	Observation

49. IF R IS NOT PERSON 1: Were you thinking about yourself or [Person 1 Name] when you marked the relationship for (NAME)?

28. Observation

What country (were you/was [NAME]) born in? COLLECT DATA FOR ALL PERSONS LISTED IN THE QUESTIONNAIRE, P1 –P6.

P49_Person 1: _______________________

P49_Person 2: _______________________

P49_Person 3: _______________________

P49_Person 4: _______________________

P49_Person 5: _______________________

P49_Person 6: _______________________

Tenure Vignette.

Now, I have a situation I would like to run past you, to see how you think someone in that situation might answer one of the survey questions: 

Mrs. Smith is responding to the census questionnaire for her household. She lives with her son, who pays the mortgage, but she doesn't pay anything. 

P45. How should she respond?  How did you come up with that answer?

P46. What does “occupied without payment of rent” mean to you in this question? Can you give me an example of when someone should pick this option?

SECTION IV. FINAL DEBRIEFING

Thanks, we’re almost finished. I have a few more questions for you.

Language fluency	Very Well	Well	Not well	Not at all
DB1. How well do you speak English?
DB2. How well do you read English?
DB3. How well do you write in English?
DB4. How well do you understand movies and videos in English?

Language use	1.Almost never	2.Some-times	3.Often	4.Almost always
DA1. How often do you think in English? Would you say…
DA2. How often do you speak English at home? Would you say…
DA3. How often do you speak English with your friends? Would you say…

FOR ALL INTERVIEWS: (Skip if interview longer than 60 minutes)

1. Aside from the things we’ve already talked about, were there any questions or instructions that you found confusing or difficult to answer?

2. Is there anything else you would like to tell me that you haven’t already?

Thank you very much for your help today. PAY R AND HAVE THEM SIGN RECEIPT. TURN OFF RECORDER.

End Time: ___________ AM / PM

[INTERVIEWER: Answer the following questions right after the interview]

1. How comfortable do you think the respondent was to answer the probe questions in general?

Not comfortable at all		Somewhat comfortable		Very comfortable
1	2	3	4	5

2. How well do you think the respondent understood the purpose of the cognitive testing?

Poor understanding		Good understanding		Excellent understanding
1	2	3	4	5

The next questions are about how the respondent answered the probes (e.g. what does x mean to you in this question? How did you come up with this answer?)

3. How frequently did you have to ask the respondent to provide more detail about their answers (eg., asking follow-up probes for clarification)?

Never	Seldom	About half the time	Usually	Always
1	2	3	4	5

4. How would you rate the respondent’s performance in answering the probe questions as intended at different stages of the cognitive interview?

4-1. Beginning (Section IV-A)

Poor job				Excellent job
1	2	3	4	5

4-2. In the middle (Section IV-B)

Poor job				Excellent job
1	2	3	4	5

4-3. Toward the end (Section IV-C)

Poor job				Excellent job
1	2	3	4	5

OMB Control Number: 0607-0725; Expiration: 8/31/16