Various Demographic Area Pretesting Activities

Date_______________________; Participant #________; Experimenter:_______________

General Introduction: Measuring Question Difficulty on the American Community Survey Internet Instrument

Thank you for your time today. My name is Rachel Horwitz and I am a student here in the Joint Program of Survey Methodology and am also employed at the US Census Bureau in the American Community Survey Data Collection area. I will be working with you today. If you have a cell phone, please turn it off or put it in vibrate. In order to help us improve our surveys, we turn to people like you to find out if our questions make sense and are fairly easy to understand and answer. We have found that the best way to do that is to actually conduct the survey with people and see how it works for them. So you will be helping us test a questionnaire from one of our surveys. Since it is difficult to recruit participants with a wide range of experience to come in to take these studies, we will be asking you to read a scenario for each question and answer the question based on the information you have read, not your own personal experience. I did not create the survey, so please share both your positive and negative reactions to it. The entire session should last approximately 45 minutes. Your comments and feedback will be given to the developers of the survey and may be used to improve it.

First, I would like to ask you to read and sign this consent form. It explains the purpose of today’s session and informs you of your rights as a participant. It also tells you that we would like to record the session, with your permission. Only those of us connected with the project will review the recording and any other data collected during the session, and it will be used solely for research purposes. We may also use clips from the recording to illustrate key points about the survey to the Web design team.

Hand the participant the consent form; give time to read and sign; sign own name and date if you have not already done so.

Start the tape.

While you are completing the survey, we will record the movements of your eyes with our eye-tracking monitor to get a record of where you are looking on the screen and we will record your mouse movements to see how you are interacting with the survey.

Now I am going to calibrate your eyes for the eye-tracking.

Do Calibration

Now that we have your eyes calibrated, we are ready to begin. Please respond to the survey questions online. Please read each scenario and then answer the questions as they apply to the scenario. To view the scenario, click the “Scenario” link at the top of the page (show screen shot of where the scenario is located).

Do you have any questions?

Start the eye-tracking software: Tobii Studio. The survey opens in Internet Explorer when Tobii is started. Tell respondents to answer as if they were home alone. Leave room.

Overall Probe: Make a note if a person left a page with a blank answer, asked a question to the researcher, or displayed signs of confusion (hovers, regressions, using the mouse as a marker).

What was your overall impression of the survey?

Were there any questions you found to be difficult or challenging to answer?

If yes, show the respondents the relevant questions again and ask them to explain what was confusing to them and what they were thinking about while answering the question.

Were there any responses you were unsure of?

If yes show the respondents the relevant questions again and ask them to explain what was confusing to them and what they were thinking about while answering the question.

If respondents reported being confused by a question or displayed signs of confusion, ask them

What was it about this question that was confusing (understanding the question, understanding the response options, applying their situation to one of the response options)?

How did you come up with your answer?

For any questions where the respondent displayed signs of confusion, ask if they had any trouble answering the question and what type of trouble they had. If they were not sure which answer category to select, ask them to describe their situation.

For any questions where the participant confirmed were difficult or they were unsure of:

If Help was not selected, Did you consider selecting the Help option?

If yes, Why did you decide not to select Help?

If Help was selected, Was the Help useful in answering the question?

Thank you again for your participation today. It is greatly appreciated.

Participant#:

RACE: White Black or African American American Indian or Alaska Native Asian Native Hawaiian or Other Pacific Islander

AGE RANGE: < 30 31-45 46-60 61+

GENDER: M F

EDUCATION: HS/GED Some Coll/AA Bachelor’s Some grad

DATE OF INTERVIEW:

INTERVIEWER:

3 | Page