Various Demographic Area Pretesting Activities

Attachment B

Date_______________________; Participant #________; Researcher: _______________

General Introduction: ACS Instrument

Thank you for your time today. My name is XX and I work here with the Census Bureau’s Human Factors and Usability group (or Census Bureau’s Population Division). I will be working with you today. If you have a cell phone, please turn it off or put it on vibrate. We will be evaluating the design of the online American Community Survey questionnaire by having you interact with it.

I did not create the survey, so please feel free to share both your positive and negative reactions to it. We are not evaluating you or your skills. Rather, you are helping us see how well the survey works. The entire session should last about an hour. Your comments and feedback will be used to improve the online survey.

First, I would like to ask you to read and sign this consent form.

Hand the participant the consent form; give time to read and sign; sign your own name and date it if you have not already done so.

It explains the purpose of today’s session and informs you of your rights as a participant. We will record your interaction with the screens using software on the computer. [Lab Sessions ONLY: It also tells you that we would like to videotape the session, with your permission. Only those of us connected with the project will review the tape and any other data collected during the session, and the information will be used solely for research purposes. We may use clips from the tape to illustrate key points about the survey to the online survey design team. In addition, observers from the project team may observe this session in another room. ]

I would like you to tell me your impressions and thoughts about the survey screens as you look at them. In other words, I would like you to “think aloud” and talk to me about your impressions. If anything was confusing on the screen, please let me know. If you expect to see some item of information let me know if you see it or not. You will interact with two versions of the survey today.

Pull up www.wtop.com in Firefox.

Before we get started, let's practice thinking aloud, since it's not something that you would normally do while working online. Pretend that you have a minute or two to kill at your desk at work or at home and talk me through your thought process as you try to find something interesting to read on this news site.

Ok, that’s exactly what I would like for you to do throughout the session. If at any time during the session you get quiet, I may remind you to talk to me. This is not to interrupt your thought process, but simply to remind you to keep talking to me. Please focus on verbalizing what you are thinking as you complete the survey.

Start the tape.

[Lab Sessions only will have eye-tracking: While you are completing the survey, we will record the movements of your eyes with our eye-tracking monitor to get a record of where you are looking on the screen.

Now I am going to calibrate your eyes for the eye-tracking.

Do Calibration

Now that we have your eyes calibrated, we are ready to begin.]

Please respond to the survey online as you would at home.

Pull up Internet Explorer.

[Lab Sessions Only: I am going to go around to the other room to do a sound check. While I am doing that, please take a moment to complete this questionnaire. [Pull up Participant background questionnaire on Computer experience and demographics]

I’m going to leave but we will still be able to communicate through a series of microphones and speakers. Do you have any questions before we begin?

Leave room. Once in control room do a sound check and start the eye-tracking software: Tobii Studio. ]

Allow the participant to proceed through the entire instrument without probing. You may answer their questions if they have trouble, but hold all probes until debriefing.

Things to watch for in order to make note of and probe about during debriefing:

• Expressions of confusion or frustration

• Expressions of satisfaction or being pleased with the outcome of interacting with the survey

• Issues with including or excluding items

• Awareness or use of help

• Attempts to change entries

• Overall ability to navigate within the web survey

• Verbal requests for help

_____________________________________________________________________________________

After the first survey version:

1. How easy or difficult was it to complete the survey?

1	2	3	4	5
Very Easy	Somewhat Easy	Neither easy nor Difficult	Somewhat Difficult	Very Difficult

2. Do you have any comments about the survey?

No	Yes, Specify______ _____________________ _____________________

3. Please complete the satisfaction questionnaire about the survey you just took.

Administer Satisfaction Qnaire 1.

4. How likely would you be to complete the survey if you received it at your home?

After the second survey version:

After the session is complete, tell the participant:

I will be asking you some more questions about your experience shortly. In the meantime, please complete this satisfaction questionnaire (Pull up Satisfaction Questionnaire).

1. How easy or difficult was it to complete the survey?

1	2	3	4	5
Very Easy	Somewhat Easy	Neither easy nor Difficult	Somewhat Difficult	Very Difficult

2. Do you have any comments about the survey?

No	Yes, Specify______ _____________________ _____________________

3. Please complete the satisfaction questionnaire about the survey you just took.

Administer Satisfaction Qnaire 2. Please only consider the second ACS survey that you completed when filling out this second satisfaction questionnaire.

After Satisfaction Qnaire 2: For the remainder of the questions, please consider both versions of the ACS survey that you completed.

4. How likely would you be to complete the survey if you received it at your home?

5. Did you notice a difference between the two versions of the Survey?

No	Yes, Specify______ _____________________ _____________________

6. The surveys had different versions of the Date of Birth Screen. Which version did you prefer, or did you have no preference? You may refer to the image here. Show paper image of both versions of the screen to the P.

No Preference

Preferred Version, Specify______ _____________________ _____________________

7. Would your preference be the same for all Date of Birth Pieces (Month, Day, and Year) or different? Why?

8. Was one version of the Date of Birth screen easier to us than the other? If so, what made it easier?

9. How comfortable would you be with answering these questions for all members of your household if you were taking this survey at home?

1	2	3	4	5
Very Comfortable	Somewhat Comfortable	Neither comfortable nor Uncomfortable	Somewhat Uncomfortable	Very Uncomfortable

10. How confident would you be in the accuracy of your answers for all members of your household if you were taking this survey at home?

1	2	3	4	5
Very Confident	Somewhat Confident	Neither Confident nor Unconfident	Somewhat Unconfident	Very Unconfident

11. Do you feel that it is easier to provide an accurate response to one version of the Date of Birth screen versus the other? If yes, which one?

12. Did you notice that age was automatically calculated after Date of Birth was entered? If so, what is your opinion of this feature of the survey?

13. Was the instruction to “Report Babies as age 0 when the child is less than 1 year old” clear?

14. If you have other people in your household, would you share the survey login information or would you have one person fill out the survey?

10. If you were completing this survey at home, would you be more likely to complete it using a computer or a tablet like an iPad?

Computer

iPad

4 | Page