Various Demographic Area Pretesting Activities

Date_______________________; Participant #________; Experimenter:_______________

General Introduction: Measuring Question Difficulty on the American Community Survey Internet Instrument

Thank you for your time today. My name is Rachel Horwitz and I am a student here in the Joint Program of Survey Methodology and am also employed at the US Census Bureau in the American Community Survey Data Collection area. I will be working with you today. If you have a cell phone, please turn it off or put it in vibrate. In order to help us improve our surveys, we turn to people like you to find out if our questions make sense and are fairly easy to understand and answer. We have found that the best way to do that is to actually conduct the survey with people and see how it works for them. So you will be helping us test a questionnaire from one of our surveys. Since it is difficult to recruit participants with a wide range of experience to come in to take these studies, we will be asking you to read a scenario for each question and answer the question based on the information you have read, not your own personal experience. I did not create the survey, so please share both your positive and negative reactions to it. The entire session should last between 30 and 45 minutes. Your comments and feedback will be given to the developers of the survey and may be used to improve it.

First, I would like to ask you to read and sign this consent form. It explains the purpose of today’s session and informs you of your rights as a participant. It also tells you that we would like to record the session, with your permission. Only those of us connected with the project will review the recording and any other data collected during the session, and it will be used solely for research purposes. We may also use clips from the recording to illustrate key points about the survey to the Web design team.

Hand the participant the consent form; give time to read and sign; sign own name and date if you have not already done so.

Start the tape.

While you are completing the survey, we will record the movements of your eyes with our eye-tracking monitor to get a record of where you are looking on the screen and we will record your mouse movements to see how you are interacting with the survey.

Now I am going to calibrate your eyes for the eye-tracking.

Do Calibration

Now that we have your eyes calibrated, we are ready to begin. Please respond to the survey questions online. Please read each scenario and then answer the questions as they apply to the scenario. To view the scenario, click the “Scenario” link at the top of the page (show screen shot of where the scenario is located).

Do you have any questions?

Start the eye-tracking software: Tobii Studio. The survey opens in Internet Explorer when Tobii is started. Tell respondents to answer as if they were home alone. Leave room.

Overall Probe: Make a note if a person left a page with a blank answer, asked a question to the researcher, or displayed signs of confusion (hovers, regressions, using the mouse as a marker).

What was your overall impression of the survey?

Were there any questions you found to be difficult or challenging to answer?

If yes, show the respondents the relevant questions again and ask them to explain what was confusing to them and what they were thinking about while answering the question.

Were there any responses you were unsure of?

If yes show the respondents the relevant questions again and ask them to explain what was confusing to them and what they were thinking about while answering the question.

Were there any questions for which you debated between two or more response options?

If yes, show the respondents the relevant questions again and ask them to explain what
was confusing to them and what they were thinking about while answering the question.

How did you decide on your answer?

If respondents reported being confused by a question or displayed signs of confusion, ask them

What was it about this question that was confusing (understanding the question, understanding the response options, applying their situation to one of the response options)?

How did you come up with your answer?

For any questions where the respondent displayed signs of confusion, ask if they had any trouble answering the question and what type of trouble they had. If they were not sure which answer category to select, ask them to describe their situation.

For attended school in the past 3 months

What does attended school mean to you?

For work questions

For difficult scenario, did you answer about one of those jobs or all of them?

For Relationship Question

Why did you answer the relationship question the way you did?

For Marital Status Question

Why did you answer the marital status question the way you did?

Retrospective Think-Aloud – pull up video of the respondent answering any questions on which they displayed signs of confusion.

I would like you to watch this recording of you answering a question and tell me what you were thinking about during each step of answering this question. If you expect to see some piece of information, tell me about that expectation as well. I may remind you to talk to me if you get quiet. Please focus on verbalizing what you were thinking as you read and answered the question.

Thank you again for your participation today. It is greatly appreciated.

Participant#:

RACE: White Black Hispanic Asian or other Pacific Islander
Native American or Alaska Native

AGE RANGE: < 30 31-45 46-60 61+

GENDER: M F

EDUCATION: HS/GED Some Coll/AA Bachelor’s Some grad

DATE OF INTERVIEW:

INTERVIEWER:

3 | Page