Various Demographic Pretesting Activities

Usability Testing of ACS Language Assistance Guides

Protocol Guide - English

Usability Testing of ACS Language Assistance Guides

Protocol Guide –English

PARTICIPANT ID #: _________________________ DATE: ____ / ____ / 2011

LANGUAGE (mark one): Chinese ____ Korean _____

INTERVIEWER NAME:

LAG CONTENT: Cover page instructions, back cover instructions, Start Here section, and Person 1 and Person 2 questions (p2) and Detailed Person questions 1-10 (p5)

START TIME: ______: ______ AM / PM

SECTION I. INTERVIEW CONSENT
	PLACE THE CONSENT FORM IN FRONT OF PARTICIPANT

A. Before we start, I would like you to read over the document in front of you. This document explains a little bit about this interview and provides information about your rights as a participant. In addition, the back page asks for your permission to have this session audio and video recorded. Please ask me any questions you have about this document. Once you have finished reading the document, please sign it.

PARTICIPANT READS AND SIGNS FORM. PROVIDE A COPY OF THE FORM TO THE PARTICIPANT

Here is a copy of this form for you to keep.

B. IF PARTICIPANT PROVIDES CONSENT TO HAVE THE SESSION AUDIO-TAPED: I will now turn on the tape recorder.

TEST TAPE RECORDER AND TURN IT ON AGAIN AFTER TEST

Now, I would like to confirm on tape that it is fine with you to record this interview.

 VERBAL CONSENT MUST BE AUDIO-RECORDED

Section I. End Time: _____ : _____ AM/PM

SECTION II. INTERVIEW OBSERVATION AND PRACTICE

A. Introduction

This is a form called “Language Assistance Guide” (LAG). It has been translated into <Chinese/Korean> for the American Community Survey (ACS) conducted by the Census Bureau.

• The Census Bureau counts population in the U.S and also conducts various kinds of surveys. The Census Bureau routinely encounter non-English speaking households and therefore would like to have materials available in other languages besides English.

• The Census Bureau wants to see how well the <Chinese /Korean> version works. We want to hear your thoughts and opinions on these materials. For example, we might ask you if you find it easy or difficult to complete the task that you are asked to do. We might also ask if there are any instructions that seem unclear or confusing.

• There are no right or wrong answers. We are simply trying to make sure the content of the LAD is clear and easy for most people who speak <Chinese/Korean> to understand and use it.

• Your participation in this review of the materials is very important because it will help the Census Bureau in improving these materials.

• In a few minutes, we will ask you to read the materials as if you were selected to participate in the American Community Survey.

• Do you have any questions before we begin?

B. Observation

INITIAL OBSERVATION BY PARTICIPANT

HOLD UP THE ENGLISH ACS QUESTIONNAIRE AND LAG AND SHOW THEM TO THE PARTICIPANT

Once a household is selected for the American Community Survey (ACS), an English-language questionnaire like this will be mailed to the household. If the household doesn’t know English well enough to complete the form, they can request a Language Assistance Guide (LAG) in <Chinese/Korean> to help them complete the ACS questionnaire.

Let’s imagine that your household has been selected to participate in the ACS, and this English form has been mailed to your household. A Language Assistance Guide in <Chinese/Korean> has also been sent to you upon your request. So now you have received these two forms: ACS, and LAG.

	PLACE THE ACS QUESTIONNAIRE, THE LAG, AND A PEN IN FRONT OF THE PARTICIPANT. INTERVIEWER: “Please look over these two forms as you would have if you had received them in the mail. As you look over these forms, please say, out loud, what you are thinking. Just say whatever comes to mind – imagine that I am not here.

INTERVIEWER OBSERVATIONS DURING INITIAL REVIEW:

• What did R do upon receiving the two forms? For example: Which did R look at/pick up first? What did R do with them? Did R skim through the materials? Read them carefully? Discard one or both of them?

• What is your assessment of R’s emotional reaction to the LAG / ACS questionnaire? For example: Positive, negative, neutral? Did R exhibit confusion, frustration, relief?

GENERAL REACTION:

SPECIFIC TO LAG:

SPECIFIC TO ACS:

• Which document did the participant spend the most time reviewing/reading?

[MARK ONLY ONE]

• The Language Assistance Guide

• The ACS questionnaire

• How did R handle this document? For example: Did R study the front cover first? Open quickly then flip back and forth? Read carefully?

• Did R read the instructions on the front of this document?

• Yes, THROUGH THE END → Document any observations made during reading:

• Yes, IN PART ONLY → Document any observations made during reading:

• No

• Did participant read the instructions on the back of the document?

• Yes, THROUGH THE END → Document any observations made during reading:

• Yes, IN PART ONLY → Document any observations made during reading:

• No

• Did participant review the other document?

• Yes, THROUGH THE END → Document any observations made during reading:

• Yes, IN PART ONLY → Document any observations made during reading:

• No

INTERVIEWER ASKS FOLLOW-UP QUESTONS BASED ON THE ABOVE OBSERVED REACTION: What drew your attention when you first looked at the forms? I noticed that you spent a lot more time looking through <ACS / LAG>. Was there any particular reason why? I notice you were [laughing/frowning/sighing…] when you saw _____. What were you thinking about at that point? I noticed you skipped this part/page, what made you do so? What were you looking at when you were on the front cover page? Is there anything specific that you recall about it? (If R was observed looking at the back cover) What were you looking at when you were on the back cover page? Do you remember anything specific?

INTERVIEWER:

SECTION II. FILLING OUT THE FORM

“Let’s start filling out the form on your own. Now, imagine that you are participating in the ACS. Please fill out the first two pages on the form as if you were doing it at home. While you are filling out the form, please verbalize any thoughts that come to mind. Describe what you are doing, as you are doing it. I will be listening, but will not be able to provide any feedback on what you are saying or doing at the moment. Again, this is NOT any kind of test and there are no right or wrong answers.

INTERVIEWER: DIRECT R TO FILL OUT THE FIRST TWO PAGES OF THE FORM (START HERE PAGE AND PERSON 1 AND PERSON 2 COLUMNS ON PAGE 2)

“Start Here” page and “Person 1” and “Person 2” page

OBSERVE HOW R FILLS OUT THE FORM AND NOTE DOWN THE ISSUES OUTLINED BELOW:

• How did R arrange the ACS and LAG? For example: Next to each other, so that the LAG could be referred to while filling out the ACS? Or did R only use one form?

• Which form did the R fill out? LAG or ACS? Use the LAG as a guide to filling out the ACS? Fill out the LAG, then transfer their answers to the ACS?

If R filled out only the LAG, did R completely ignore the ACS form?

• Or did R begin filling out the ACS form, then eventually forgot about it?

• Did R appear to understand the questions, but have trouble finding the appropriate answer box in the English questionnaire?

Did R fill out the write-in spaces using their native script or using English letters, or both?

• If R wrote answers in both their native script and English, Did R write some answers in native script, others in English script? For example: R writes name in native script, but other entries in English script?

• If R filled out the LAG, then transferred their answers, did the script change? How did it change? For example: R wrote in their native script on the LAG, but wrote the corresponding answers on the ACS in English.

ASK THE FOLLOWING QUESTONS AS NEEDED TO FOLLOW UP ON EACH OF THE OBSERVATIONS ABOVE:

• “Now that you have filled out the form, imagine that you are now going to return it by mail. Which form would you return in the mail?”

“I noticed that you put your answers in the LAG, but not in the English form. What were you thinking at that time?”

• “I noticed you wrote your answers in < Chinese/Korean >, what made you decide to do so? ”

• “I noticed you were ...., can you please tell me what led you to do …?”

INTERVIEWER: AFTER R COMPLETES QUESTIONS ON PAGE 2, DIRECT R TO GO TO THE FRONT PAGE, LOOK AT THE INSTRUCTIONS ON THE FRONT PAGE AND ASK THE FOLLOWING PROBES.

Instruction on the front page of the LAG

INTERVIEWER: DIRECT R TO RE-READ THE BULLETED INSTRUCTIONS ON THE FRONT PAGE OF THE LAG “Now, I would like you to read these lines again and let me know when you are done reading.”

POINT TO THE FIRST BULLET IN INSTRUCTIONS

• “Can you tell me what you think is meant by this statement?”

POINT TO THE SECOND BULLET IN INSTRUCTIONS

• “Can you tell me what this instruction asks you to do?”

POINT TO THE THIRD BULLET IN INSTRUCTIONS

• “Can you tell me what this instruction asks you to do?”

• IF R SAID “MAIL BACK THE FORM”, ASK:

• “Which form should you mail back?”

INTERVIEWER: AFTER R COMPLETED FRONT PAGE AND QUESTIONS ON PAGE 2, DIRECT R TO TURN TO PAGE 8 AND TO COMPLETE QUESTIONS 7-15 ON DETAILED PERSON 1 SECTION. OK, now that you have read the instructions carefully for how to complete the ACS questionnaire, let’s turn to page 8 and complete the questions on page 8. For this portion of the form, I would like you to follow the instructions that we discussed just now. That is, look at the questions in the (CHINESE/KOREAN) language assistance guide, and then answer the questions in the English form. INTERVIEWER: OBSERVE AND NOTE DOWN THE DETAILED ISSUES OUTLINED BELOW:

Detailed Person Questions 7-15 (page 8)

• Did R fill out the LAG rather than the ACS form?

• If so, is it because they forget about the ACS form or because they don’t know English well enough to find the corresponding answer box?

• Did R understand the questions, but had trouble finding the appropriate answer box in the English questionnaire?

• Did R follow the skip instructions?

• Could R follow the appropriate skip pattern and find the right questions/answers on the ACS questionnaire?

• Did R fill out the write-in spaces using their native script or using English letters?

• ASK THE FOLLOWING QUESTONS AS NEEDED TO FOLLOW UP ON EACH OF THE OBSERVATIONS ABOVE:

• “I notice that you put your answer in the LAG, but not in the English form, what were you thinking at that time?”

• “I notice you wrote your answers in Chinese/Korean, what made you decide to do so? ”

• “I notice you were ...., can you please tell me what led you to do …?”

INTERVIEWER: AFTER R COMPLETES QUESTIONS ON PAGE 8, DIRECT R TO GO TO THE ROUTING INSTRUCTION AT THE END OF PAGE 11 AND AT THE BEGINNING OF PAGE 12. ASK THE FOLLOWING PROBES. POINT TO THE ROUTNING INSTRUCTION ON PAGE 11 “Can you tell me what you think is meant by this statement?” “What should you do now?” POINT TO THE INSTRUCTION ON PAGE 12 “Can you tell me what you think you should do next according to this instruction?”

	INTERVIEWER: DIRECT R TO GO TO THE LAST PAGE (P16) OF LAG AND READ THE INSTRUCTIONS ON P16. ASK THE FOLLOWING PROBES. “Now that you have completed the questionnaire, can you tell me what you should do now according to this instruction?” IF R SAID “MAIL BACK THE FORM”, ASK: “Where are you going to mail the form?” “What will you use to mail it back?”

Debriefing questions

• OK, now that you have completed the questionnaire, can you tell me if it was easy or difficult it was for you to look at a Chinese/Korean language guide and fill out the answers in the English questionnaire? Why?

• Which parts were most confusing or problematic?

• What do you think of the entire experience/process?

Closing questions

Finally, I have just a few more questions.

Q1. Have you ever filled out an English form by looking at a CHINESE/KOREAN language assistance guide like what we did today?

1 Yes  Tell me more about it. What was it like? By what organization?

2 No

Q2. Have you ever participated in any interview like the one we did today?

1 Yes  Tell me more about it. What was it like? By what organization?

2 No

Q3. Have you filled out a survey questionnaire or a similar form in your country of origin?

1 Yes  Tell me more about it. What kind? Was it for market research, demographic surveys, or something else?

2 No

Q4. How about in the U.S.? Have you filled out a survey questionnaire or a similar form in the U.S.?

1 Yes  Tell me more about it. What kind? Was it for market research, demographic surveys, or something else?

2 No

Q5. Have you ever participated in any survey interview before? For example, instead of your filling out the answers to questions on a survey questionnaire like today, the survey questions were read to you by an interviewer? This can be done in person or over the phone.

1 Yes  Tell me more about it. Was it by phone or in person? What was it like? By what organization?

2 No

N_Debr1. If you receive this form at home by mail and it is written in English. What would you do? (Will you try to fill it out by yourself or will you ask someone else for help? Who would that be?)

1 By her/himself

2 Somebody else (Specify:              )

N_Debr2. What if the form is written in [Korean/Chinese]?  Will you (try to) fill out the form by yourself or will you ask for help to somebody else?

1 By her/himself

2 Somebody else (Specify:              )

Q6. Do you have any final comments or questions?

I want to thank you very much for your participation. I will now give you $40 and I will ask you to sign a receipt form verifying that you received the money.

INTERVIEWER: TURN OFF THE TAPE RECORDER. HAND THE CASH INCENTIVE TO THE PARTICIPANT.

End Time: ____________ AM / PM

INTERVIEWER ONLY: ANSWER THESE QUESTIONS AFTER THE INTERVIEW.

N_Debr3. HOW DIFFICULT WAS IT FOR YOU TO CONDUCT THE INTERVIEW WITH THIS RESPONDENT?

1 Very difficult 2 Somewhat difficult 3 Somewhat easy 4 Very easy

N_Debr4. WHAT FACTOR DO YOU THINK ATTRIBUTE TO THIS RESPONDENT’S COGNITIVE INTERVIEW MOST?

1 Age 2 Education 3 Others (Specify: _____________________________________)

10

CSM Project 02-25-2011