Various Pretesting Activities

Cognitive Testing of Translated ACS Language Assistance Guides

Protocol Guide - English

Cognitive Testing of Translated ACS Language Assistance Guides

Protocol Guide – English

“B”

Phase 1 - Round 1

PARTICIPANT ID #: |_P_|___|_R_|___|___|___|___|___|___| DATE: ____ / ____ / 2010

LANGUAGE (CHECK ONE): 1 CHINESE  Specify dialect: ____________________

2 KOREAN

INTERVIEWER: 1 Ginny 2 Hyunjoo 3 Jiyoung 4 Suzie 5 Younhee

6 Gretchen 7 Guanya 8 Mandy 9 Michelle 10 Lu 11 Grace

LAG CONTENT: Start Here box, Persons 1 and 2, and Housing Questions 12 – 21

START TIME: ____________ AM / PM

SECTION I: INFORMED CONSENT

PLACE THE CONSENT FORM IN FRONT OF PARTICIPANT

A. Before we start, I would like you to read over the document in front of you. This document explains a little bit about this interview and provides information about your rights as a participant. In addition, the back page asks for your permission to have this session audio recorded. Please ask me any questions you have about this document. Once you have finished reading the document, please sign it.

PARTICIPANT READS AND SIGNS FORM. PROVIDE A COPY OF THE FORM TO THE PARTICIPANT

Here is a copy of this form for you to keep.

B. IF PARTICIPANT PROVIDES CONSENT TO HAVE THE SESSION AUDIO-TAPED: I will now turn on the tape recorder.

TEST TAPE RECORDER AND TURN IT ON AGAIN AFTER TEST

Now, I would like to confirm on tape that it is fine with you to record this interview.

 VERBAL CONSENT MUST BE AUDIO-RECORDED

SECTION II. INTERVIEW OBSERVATIONS

This is a form called “Language Assistance Guide”. It has been translated in [Chinese/Korean] for the American Community Survey conducted by the Census Bureau.

• The Census Bureau counts population in the U.S. and also conducts various kinds of

surveys. They routinely encounter non-English speaking households and therefore would

like to have this form available in other languages beside English. It will be mailed to

people who request them across the U.S.

• Researchers at the Census Bureau are interested in knowing your opinions about the translation.

• They have hired RTI International to conduct this research. Your participation is very

important because it will help the Census Bureau determine if this form is translated well

and is culturally appropriate.

• I’d like you to read through this form and record your answers to the questions. Please try to complete the form on your own, as if I were not here. While you fill out the form, I’ll be taking some notes and then we will discuss the translation together.

• This is NOT any kind of test and there are no right or wrong ways to fill it out. We are practicing with as many different kinds of people as possible in different parts of the country to see if everyone fills out the form the same way and to see if it’s easy or difficult for people to fill out.

Here is the form. When you are ready, begin with “Start Here”.

INTERVIEWER: For Protocol “B”, direct respondents to complete the Start Here box, Person 1, Person 2 (if applicable), and Housing questions 12-21. They should also read routing instruction E but do not need to continue in the form. Note perceived difficulty and errors experienced by the respondent. Discuss these issues during the cognitive interview and probe on translation issues. Sign of confusion or difficulty (reading but not answering, asking questions, etc.) Skipped questions intentionally or unintentionally Didn’t follow instructions (checking wrong boxes, following wrong path, etc.) If R asks questions, encourage R to complete the form on his or her own. Say: What would you do if I weren’t here? End this section if the participant becomes upset or is unable to complete the form on his or her own after 15 minutes.

A. Introduction and Practice

Now I have some specific questions about how you understood some of the words and phrases that were used in this form. I will first point out these words and phrases to you and let you read them over before I ask about your thoughts. I really want to hear all of your opinions and reactions.

Let’s do a short practice before we begin. Please look at this question:

How many windows are there in the house or apartment where you live?

What would your answer be?

Now, tell me everything you were thinking about as you went about answering this question.

INTERVIEWER: Probe as necessary.

• How did you come up with that answer?

• What do you think they mean by the term “window”?

• What is this question asking in your own words?

Thank you. Don’t hesitate to speak up whenever something seems unclear, is hard to answer, or doesn’t seem to apply to you. Again, this is NOT any kind of test and there are no right or wrong answers. Just tell me what you think of the translation.

Okay, now let’s get to the form that you filled out.

SECTION III. COGNITIVE INTERVIEW

B. Housing Questions 12-21

Housing Question 12

First, take a look at Question 12. How did you come up with your answer?

INTERVIEWER: ADMINISTER PROBES AND RECORD ANSWERS IN TABLE BELOW.

This question mentions several government programs.

• Have you heard of [PHRASE] before? (IF APPLICABLE, SAY: Its English acronym is …, have you hard of it before?)

• What do you think they mean by [PHRASE] here?

• [IF YES] Is there a better way to describe this program?

Phrase	Heard before? (y/n)	(If heard before) Better translation	What it is
Food Stamps
Food Stamp benefit card
Supplemental Nutrition Assistance Program (SNAP)
WIC
National School Lunch Program

Housing Question 13

In Question 13, what do you think they mean by “condominium” here? Have you heard of the English word “condo” or “condominium”?

What do you think they mean by “condominium fee”?

The “condominium fee” is assessed by the condominium owners’ association for the purpose of improving and maintaining the common areas. Are there better ways to describe it?

Now I’d like you to read the instruction here [POINT TO ITALIC TEXT] aloud to me then I will ask some questions. “For renters, answer only if you pay the condominium fee in addition to your rent; otherwise, mark the “None” box.”

INTERVIEWER: WHILE R IS READING, DOCUMENT DIFFICULTIES OBSERVED. PLACES WHERE TEXT/SYNTAX SEEMS DIFFICULT FOR RS TO READ AND/OR COMPREHEND. ANY TERMS/CONCEPTS MENTIONED BY RS AND THE ISSUES THEY HAVE WITH THOSE TERMS/CONCEPTS. ANY MISPRONUNCIATIONS. (THIS WILL SERVE AS A CLUE FOR DIFFICULT OR UNKNOWN TERMS/CONCEPTS.) PROBLEMS WITH QUESTION FORMAT

• PROBE IF APPROPRIATE: “I noticed when you read those instructions…, you were (laughing/ hesitating/pausing), what were you thinking at that point?”).

• Do those instructions sound natural to you in <language>?

• What is the instruction saying in your own words?

INTERVIEWER: DISCUSS DIFFICULTIES WITH QUESTION LAYOUT IN QUESTION 13.

Routing Instruction B

Now please read the phrase right above Question 15 to yourself. What do you think these phrases are asking you to do?

Did you notice them while filling out the form?

INTERVIEWER: IF R DID NOT FOLLOW THE ROUTING INSTRUCITON, DETERMINE THE CAUSE. Tell me how you decided to answer Question #. Tell me how you decided to skip Question #.

Housing Question 15a-b

You are doing great. Let’s take a look at Question 15b. IF R SKIPPED QUESTION 15b, SAY: That’s okay. Let’s look at it now.

What do you think this question is asking here?

Routing Instruction C

Now please read the phrase right above Question 16 to yourself. What do you think these phrases are asking you to do?

Did you notice them while filling out the form?

INTERVIEWER: IF R DID NOT FOLLOW THE ROUTING INSTRUCITON, DETERMINE THE CAUSE. Tell me how you decided to answer Question #. Tell me how you decided to skip Question #.

Housing Question 16

Let's look at Question 16. IF R SKIPPED QUESTION 16, SAY: That’s okay. Let’s look at it now.

Was there something confusing? What is this question asking in your own words?

What does the term “lot” mean to you here?

Housing Questions 17

Please read Questions 17 to yourself. IF R SKIPPED IT, SAY: That’s okay. Let’s look at it now.

What do you think they mean here in Question 17?

How did you come up with the amount?

When you read the phrase “annual real estate taxes” in this question, what period of time were you thinking about?

What does the term “property” mean to you here?

INTERVIEWER: DISCUSS DIFFICULTIES WITH QUESTION LAYOUT IN QUESTION 17.

Housing Questions 18

Please read Questions 18 to yourself. IF R SKIPPED IT, SAY: That’s okay. Let’s look at it now.

What do you think they mean here in Question 18?

How did you figure the amount?

When you read the phrase “annual payment for fire, hazard, and flood insurance” in this question, what period of time were you thinking about?

What does the term “none” mean to you here?

Housing Question 19a and 19b

Okay, let's look at Question 19a. How did you come up with your answer?

IF R SKIPPED QUESTION 19a, SAY: That’s okay. Let’s look at it now. What do you think this question is asking?

Have you heard of the term “deed of trust”?

A deed of trust is used in some areas instead of a mortgage and involves a third party, known as the trustee, who is given technical title to the property. Can you think of a better way to say this phrase in your language?

Have you heard of the term “contract to purchase”?

With a contract to purchase, the original owner has the title to the property, while the borrower has the right to occupy or otherwise use the property so long as he or she makes the required payments to the original owner towards the purchase of the property. Can you think of a better way to say this phrase in your language?

Now, please read Question 19b aloud. What do you think this question is asking about?

How did you come up with your answer?

In this question, what period of time were you thinking about?

What do you think they mean by “regular mortgage payment?”

What do you think they mean by “Include payment only on FIRST mortgage or contract to purchase”.

Housing Question 20a-b

Okay, let's look at Question 20a and 20b. How did you come up with your answer?

IF R SKIPPED QUESTION 20a-b, SAY: That’s okay. Let’s look at it now. What do you think this question is asking?

Have you heard of the term “second or junior mortgages”?

A second or junior mortgage is secured by real estate but has been made by the homeowner in addition to the first mortgage. It gives the lender a claim against the property which is second to the claim of the holder of the first mortgage. Can you think of a better way to say this phrase in your language?

Housing Question 21

Good, now Question 21. IF R SKIPPED QUESTION 21, SAY: That’s okay. Let’s look at it now. What do you think this question is asking?

There are several terms in Question 21. Let’s discuss them. First, could you figure out what the phrase “personal property taxes” mean here?

This question is about a mobile home. It is a home that can be moved around. It is parked on a piece of land and there may be rent for using the land. The translation here says “site rent”. Does it sound natural to you for describing this land?

Routing Instruction E

[POINT TO INSTRUCTION E] Now please read these phrases. Did you notice them while filling out the form?

What do you think these phrases are asking you to do?

C. Debriefing and Closing

Finally, I have just a few more questions.

1. Have you ever participated in any cognitive interviewing like the one we did today?

1 Yes  Tell me more about it. By what organization?

2 No

2. Have you filled out a survey questionnaire in your country of origin?

1 Yes  Tell me more about it. Was it for market research, demographic surveys, or something else?

2 No

3. How about in the U.S.? Have you filled out a survey questionnaire in the U.S.?

1 Yes  Tell me more about it. What kind? Was it for market research, demographic surveys, or something else?

2 No

4. Have you ever participated in any survey interview before? For example, instead of your filling out the answers to questions on a survey questionnaire like today, the survey questions were read to you by an interviewer? This can be done in person or over the phone.

1 Yes  Tell me more about it. Was it by phone or in person? What was it like? By what organization?

2 No

5. Think about your family members, friends and neighbors who speak [Chinese/Korean] primarily. Have you heard that they have ever participated in any survey interview before?

1 Yes

2 No

INTERVIEWER: MAKE ADDITIONAL NOTES HERE.

6. INTERVIEWER: PROBE FOR THE OBSERVED DIFFICULTIES IN SECTION II THAT

YOU HAVE NOT DISCUSSED WITH THE RESPONDENTS. I noticed when you were filling out the questionnaire, you ….., what were you thinking at that point?

7. Are there any phrases or words in this form that did not sound natural to you but we have not discussed? What about things that would not be appropriate for you or someone else who speaks [Chinese/Korean]? For example, offensive or strange wording, etc.

8. [TIME PERMITTING] Let’s look at the Start Here box again. I’m going to read a brief description of a Mr. Yang’s situation: Mr. Yang’s daughter goes to school in another state. She lives in the dormitory during the academic year but spends summer vacation with Mr. Yang. According to the DO NOT INCLUDE instructions, should Mr. Yang count his daughter in this question if he is filling it out during the summer? Why or why not?

9. Do you have any final comments or questions?

I want to thank you very much for your participation. I will now give you $40 and I will ask you to sign a receipt form verifying that you received the money.

INTERVIEWER: TURN OFF THE TAPE RECORDER. HAND THE CASH INCENTIVE TO THE PARTICIPANT.

End Time: ____________ AM / PM

10

RTI Project 0209182.017