Various Demographic Pretesting Activities

Usability Testing of ACS Version ACS-1(X)QD85

Protocol Guide

Usability Testing of ACS Version ACS-1(X)QD85

Protocol Guide

PARTICIPANT ID #: _________________________ DATE: ____ / ____ / 2012

INTERVIEWER NAME:

START TIME: ______: ______ AM / PM

SECTION I. INTERVIEW CONSENT
	PLACE THE CONSENT FORM IN FRONT OF PARTICIPANT

A. Before we start, I would like you to read over the document in front of you. This document explains a little bit about this interview and provides information about your rights as a participant. In addition, the back page asks for your permission to have this session audio recorded. Please ask me any questions you have about this document. Once you have finished reading the document, please sign it.

PARTICIPANT READS AND SIGNS FORM. PROVIDE A COPY OF THE FORM TO THE PARTICIPANT

Here is a copy of this form for you to keep.

B. IF PARTICIPANT PROVIDES CONSENT TO HAVE THE SESSION AUDIO-TAPED: I will now turn on the tape recorder.

TEST TAPE RECORDER AND TURN IT ON AGAIN AFTER TEST

Section I. End Time: _____ : _____ AM/PM

SECTION II. FILLING OUT THE FORM

A. Introduction

[SHOW RESPONDENT ACS FORM] This is a form called the American Community Survey (ACS), and is conducted by the Census Bureau.

• The Census Bureau counts the U.S. population and also conducts various kinds of surveys.

• The Census Bureau wants to see how well the ACS works. We want to hear your thoughts and opinions on this form.

• There are no right or wrong answers. We are simply trying to make sure the content of this form is clear and easy for most people to understand and use it.

• Your participation in this review of this survey is very important because it will help the Census Bureau improve the ACS.

• In a few minutes, we will ask you to fill out the form as if you were selected to participate in the American Community Survey.

• Do you have any questions before we begin?

B. Observation

INTERVIEWER: PLACE THE ACS QUESTIONNAIRE AND A PEN IN FRONT OF THE PARTICIPANT.

Let’s start filling out the form on your own. Now, imagine that you are participating in the ACS. Please complete the whole form from beginning to end as if you were doing it at home. I will be listening, but will not be able to provide any feedback on what you are saying or doing at the moment. Again, this is NOT any kind of test and there are no right or wrong answers.

INTERVIEWER: AS THE RESPONDENT FILLS OUT THE FORM, MAKE THE FOLLOWING OBSERVATIONS ASSOCIATED WITH EACH SECTION BELOW. WHEN THE RESPONDENT FINISHES COMPLETING THE FORM, ASK THE PROBES ASSOCIATED WITH EACH SECTION:

INTERVIEWER OBSERVATIONS DURING RESPONDENT RECEIVES FORM:

• What did R do upon receiving the form? For example: Which did R look at first? What did R do with it? Did R skim through it? Read it carefully? Open quickly then flip back and forth? Read carefully? Did R comment on the length of the form?

• What is your assessment of R’s emotional reaction to the ACS questionnaire? For example: Positive, negative, neutral? Did R exhibit confusion, frustration, relief?

GENERAL REACTION:

“Start Here” page and “Persons 1-12” pages

OBSERVE HOW R FILLS OUT THE FORM AND NOTE DOWN THE ISSUES OUTLINED BELOW:

• Did R notice the Internet option? Did R say he/she would prefer to use Internet?

• Did R have problems with entering names or races in the write-in boxes?

INTERVIEWER: DURING THE DEBRIEFING, ASK THE FOLLOWING QUESTONS AS NEEDED TO FOLLOW UP ON EACH OF THE OBSERVATIONS ABOVE:

• “I noticed you were ...., can you please tell me what led you to do …?” IF NECESSARY, POINT TO THE INSTRUCTION.

Skip Instructions on Housing Questions (Pages 5-8)

OBSERVE HOW R FILLS OUT THE FORM AND NOTE DOWN THE ISSUES OUTLINED BELOW:

• Did R appear to understand skip instructions “A” through “E” for the Housing questions section and find the right questions/answers? If not, what instructions did the R find problematic? What was the R’s reaction?

• Did R seem to understand the difference between questions 14 a and b on PAGE 6 which ask about monthly utility costs and 14 c and d on PAGE 7 which ask about yearly utility costs? If not, does it appear that 14 c and d are also monthly costs?

INTERVIEWER: DURING THE DEBRIEFING, ASK THE FOLLOWING QUESTONS AS NEEDED TO FOLLOW UP ON EACH OF THE OBSERVATIONS ABOVE:

• “I noticed you were ...., can you please tell me what led you to do …?” IF NECESSARY, POINT TO THE INSTRUCTION.

INTERVIEWER: DURING THE DEBRIEFING, DIRECT R TO RE-READ SKIP INSTRUCTION “C” IN THE RIGHT COLUMN ON PAGE 7. “Now, I would like you to read these lines again and let me know when you are done reading.”

POINT TO SKIP INSTRUCTION “C.”

• “Can you tell me what this instruction asks you to do?”

Detailed Person Questions for Persons 1 – 5 (Pages 9 – 43)

OBSERVE HOW R FILLS OUT THE FORM AND NOTE DOWN THE ISSUES OUTLINED BELOW:

• Did R appear to understand skip instructions “F” through “L” for the Detailed Person questions section and find the right questions/answers? If not, what instructions did the R find problematic? What was the R’s reaction?

• If R had a bachelor’s degree and answered question 12 on PAGE 10, did they miss answering question13?

• Did R think that the bottom of Page 11 was the end of the questions for Person 1? Did R have a reaction to the fact there are additional questions for Person 1 on Pages 12-15?

INTERVIEWER: DURING THE DEBRIEFING, ASK THE FOLLOWING QUESTONS AS NEEDED TO FOLLOW UP ON EACH OF THE OBSERVATIONS ABOVE:

• “I noticed you were ...., can you please tell me what led you to do …?” IF NECESSARY, POINT TO THE INSTRUCTION.

INTERVIEWER: DURING THE DEBRIEFING, DIRECT R TO RE-READ SKIP INSTRUCTION “K” AT THE MIDDLE OF THE RIGHT COLUMN ON PAGE 13. “Now, I would like you to read these lines again and let me know when you are done reading.”

POINT TO SKIP INSTRUCTION “K.”

• “Can you tell me what this instruction asks you to do?”

INTERVIEWER: DURING THE DEBRIEFING, DIRECT R TO LOOK AT PAGE 12 AND PAGE 13 SO THAT THE RESPONDENT CAN COMPARE THE LAYOUT OF THESE TWO PAGES. “Now, I would like you to look at these pages again for a few moments.”

• What do you think about the amount of space at the bottom of Page 12 compared to the amount of space at the bottom of Page 13? Would you like to have pages more like Page 12 or more like Page 13 when filling out the form? Why?

SECTION III. FORM COMPARISON

INTERVIEWER: NOW SHOW THE RESPONDENT THE OTHER FORM (FORM 36)

And now take a look at this other form [give a few moments for R to read the form].

What are some of the most noticeable differences between this and the form you saw first? Anything else?

[Show R the 3-column Housing Questions of PAGE 5 on FORM 36]. What do you think about the three columns of questions for this form compared to the two columns for the form you filled out? Would you prefer to have these questions in three columns or two? Why?

Between these two forms, which one do you like best? Why?

Debriefing questions

INTERVIEWER: NOW ASK THE RESPONDENT FOR THE COMPLETED ACS FORM FOR YOU TO REVIEW AND CONFIRM YOUR NOTED OBSERVATIONS. THEN HAND THE ACS FORM BACK TO THE RESPONDENT.

• OK, now that you have completed the questionnaire, what drew your attention when you first looked at the form?

• I noticed that you spent a lot more time looking through and particular part of the form. Was there any particular reason why?

• I notice you were [laughing/frowning/sighing…] when you saw _____. What were you thinking about at that point? I noticed you skipped this part/page, what made you do so?

INTERVIEWER: NOW GO BACK AND ASK FOLLOW UP QUESTIONS AND PROBES BASED ON THE OBSERVATIONS YOU MADE, STARTING WITH OBSERVATIONS ON “START HERE” AND “PERSONS 1-12 PAGES.”

• Can you tell me if it was easy or difficult it was for you to fill out the answers in the questionnaire? Why?

• Which parts were most confusing or problematic? For Respondents with more than one person in the household, point to the top of Page 9 and then the top of page 16: Person 1 starts on the right page and Person 2 starts on the left page. What do you think about these questions for Person 1 and Person 2 starting on opposite pages?

• What do you think of the entire experience/process?

• Do you have any final comments or questions?

I want to thank you very much for your participation. I will now give you $40 and I will ask you to sign a receipt form verifying that you received the money.

INTERVIEWER: TURN OFF THE TAPE RECORDER. HAND THE CASH INCENTIVE TO THE PARTICIPANT.

End Time: ____________ AM / PM

INTERVIEWER ONLY: ANSWER THESE QUESTIONS AFTER THE INTERVIEW.

N_Debr3. HOW DIFFICULT WAS IT FOR YOU TO CONDUCT THE INTERVIEW WITH THIS RESPONDENT?

1 Very difficult 2 Somewhat difficult 3 Somewhat easy 4 Very easy

N_Debr4. WHAT FACTOR DO YOU THINK ATTRIBUTE TO THIS RESPONDENT’S COGNITIVE INTERVIEW MOST?

1 Age 2 Education 3 Others (Specify: _____________________________________)

9

CSM Project 07-24-2012