Various Demographic Pretesting Activities

Date_______________________; Participant #________; Edit Check (1 or 2)

General Introduction: American Community Survey Internet Instrument

(English Usability Test)

Thank you for your time today. My name is XX and I work here with the Human Factors and Usability group. I will be working with you today. We will be evaluating the design of the online American Community Survey by having you complete it [for PRCS: in Spanish]. Your experience with the survey is an essential part of our work. I did not create the survey, so please share both your positive and negative reactions to it. We are not evaluating you or your skills, but rather you are helping us see how well the survey works. The entire session should last about an hour. Your comments and feedback will be given to the developers of the survey and may be used to improve it.

First, I would like to ask you to read and sign this consent form. It explains the purpose of today’s session and informs you of your rights as a participant. It also tells you that we would like to videotape the session, with your permission. Only those of us connected with the project will review the tape and any other data collected during the session; and it will be used solely for research purposes. We may also use clips from the tape to illustrate key points about the survey to the Web design team. In addition, there may also be observers from the project team observing this session in another room.

Hand the participant the consent form; give time to read and sign; sign own name and date if you have not already done so.

Start the tape.

I would like you to tell me your thoughts about the screens as you look at them. In other words, I would like you to ``think aloud'' as you go through the Web pages. If you expect to see a piece of information and it isn’t there, tell me. If you expect something to happen and it doesn’t happen, tell me. Let me know if you don’t know what to do next. Finally, if anything is confusing on the screen, let me know that too.

Pull up www.wtop.com in Firefox.

Before we get started, let's practice thinking aloud, since it's not something that you would normally do while working online. Pretend that you have a minute or two to kill at your desk at work or at home and talk me through your thought process as you try to find something interesting to read.

Ok, that’s exactly what I would like for you to do throughout the session. If at any time during the session you get quiet, I may remind you to talk to me. This is not to interrupt your thought process, but simply to remind you to keep talking to me. Please focus on verbalizing what you are thinking as you complete the survey.

Do you have any questions about the think aloud technique that we just practiced?

While you are completing the survey, we will record the movements of your eyes with our eye-tracking monitor to get a record of where you are looking on the screen and we will record your mouse movements to see how you are interacting with the survey.

Now I am going to calibrate your eyes for the eye-tracking.

Do Calibration

Now that we have your eyes calibrated, we are ready to begin. Please respond to the survey online as you would at home. You may answer the survey questions as they apply to you in your real life. Although the materials will give you an internet address, or URL, to enter to access the survey, you will not need to enter that because our testing software will open the survey for you.

Please imagine that you received this packet in the mail at your home. These are the mailing materials you will need to start the survey. If you were to receive the survey at your home, the mailing materials would have your real address. Since we cannot replicate that for the lab setting, all participants will use the same address. For the purposes of this study, please pretend that your address is 198 Young Rd in Anytown MD.

I am going to go around to the other room to do a sound check. While I am doing that, please take a moment to complete this questionnaire. [Hand Participant questionnaire on Computer experience and demographics]

I’m going to leave but we will still be able to communicate through a series of microphones and speakers. Do you have any questions before we begin?

Leave room. Once in control room do a sound check and Start the eye-tracking software: Tobii Studio. The mouse tracing software will start when Studio opens Internet Explorer.

Encourage R to think aloud while completing the survey. Ask probe questions about what they are thinking if they are having trouble with any part of the survey.

A

ddress Verification: Clicked on help []

L

ive or stay (you or anyone): Clicked on help []

F

ocus 5. PIN Screen: Seemed to read it [] Wrote PIN down []

Respondent name and phone Number: Clicked on help []

Roster A (everyone who is living or staying here):

Observe: Clicked on help []

Focus 3. Observe: Is there a problem with the position of roster list? [] Yes-Explain below [] No

Focus 4. Observe any problems with respondent name prefilled. [] Yes-Explain below [] No

D

ebrief: Did you notice that your name was already entered on the first line?

D

ebrief: Did you notice the link “Click here to add more people”?

D

ebrief: What do you think would be displayed when you clicked it?

Roster B (roommates, foster children, boarders):

Clicked on help []

F

ocus 3.Observe: Is there a problem with the position of roster list? [] Yes-Explain below [] No

• Add_1 They only get this if they said “yes” to roster_b so this may not be relevant to R.

D

id you notice that a follow-up question appeared at the bottom of the question that you just answered? Was this confusing?

I

f any names were entered on Roster b, ask: Why didn’t you include this person/these people on the original list?

Roster C (anyone there for a short time):

Clicked on help []

F

ocus 3. Observe: Is there a problem with the position of roster list? [] Yes-Explain below [] No

D

ebrief: If any names were entered, ask: Why didn’t you include this person/these people on the original list?

Screening Questions

Away Now (anyone on the list who is away for more than 2 months) and Selection Screen:

Clicked on help []

Debrief: How would you answer this question if you had a child who is currently attending and living at a high school-level boarding school?

Debrief: [] If any names were selected, ask: Why did you select this person/these people?)Was anything in this question confusing?

D

ebrief: [] If no names were selected: In your opinion, what are some examples of situations where someone would be away for more than two months.

Another Place where they usually stay/Selection Screen/Away for more than 2 months:

Clicked on help []

Debrief: What do you think is meant by “another place where they usually stay”?

Debrief: How would you answer this question if you had a child in a shared custody situation where the child usually lives with you, but is staying with the other parent on the day you complete the survey?

Debrief: How would you answer this question if you had a child in a shared custody situation where the child usually lives with the other parent, but is with you on the day you complete the survey?

Final list of eligible people:

Clicked on help []

D

ebrief: If any names were selected on the previous screens, ask: Did you notice that the names you selected on the previous screen are not on this list?

(Focus 1) Awareness or use of progress indicator

I’m going to pause you for a second:

About how much longer do you think the survey will take? How did you come up with that answer?

Demographics:

Reference Person:

Debrief: Was this question confusing? If yes, Why?

D

ebrief: Did you realize you could mark more than one person, if necessary?

Relationship:

Clicked on help []

Sex

Clicked on help []

DOB and Age [] R knew all dobs [] R didn’t know at least one dob

Clicked on help []

Debrief: if R knew dob for everyone: What would you do if you didn’t know someone’s date of birth?

Debrief: if R knew dob for everyone: What would you do if you knew the month but not the year?

D

ebrief: if didn’t get edit: Please enter an age of 29 and not a date of birth and then hit next. What does the message mean in your own words. What would you do at this point?

Hispanic Origin

Clicked on help []

Observe: Does the respondent write in a race category in the other box? (y/n)

Race

Clicked on help []

Observe: [] R knows to scroll down to get Next button []Rhas to be told to scroll down

Observe [] R put Hispanic in Other field [] R marked more than one race

Housing Screens:

Type of Unit

Clicked on help []

Year built

Clicked on help []

Moved in

Observe: Did R try to put letters in month field? If so, what did they think the error meant? Did R enter a 4 digit or 2 digit year?

# of acres

Clicked on help []

Agriculture question [] question didn’t occur

Business question

Clicked on help []

# rooms/#bedrooms

Clicked on help []

Observe: did R have trouble entering the answer in the boxes

Facilities

Clicked on help []

Observe: did R put an answer in each row of the table? Or just mark the “yes” answers?

Autos

Clicked on help []

Heating fuel

Clicked on help []

Pay for electricity (y/n)

Clicked on help []

Electric Amount: amount entered = [ ]

Clicked on help []

Observe: Did R enter amount correctly? Did R put “$”? did R put enter decimal amount? If they do this can we debrief on it?

Debrief: Are your gas and electricity billed together or separately. (If together), does the amount you entered include both?

Included electricity: [] in rent fee OR [] no charge

Use gas

Pay for gas (y/n)

Clicked on help []

Pay for gas: amount entered = [ ]

Clicked on help []

Observe: Did R enter amount correctly? Enter a comma, decimal or $ sign?

Debrief: Are your gas and electricity billed together or separately. (If together), does the amount you entered include both?

Included gas: [] in rent fee [] in electricity bill OR [] no charge

Pay for water/sewer (y/n)

Clicked on help []

Pay for water/sewer: amount entered = [ ]

Observe: Did R enter amount correctly?

Clicked on help []

Included water/sewer: [] in rent fee OR [] no charge

Use other fuel sources

Pay for these sources (y/n)

Clicked on help []

Pay for other sources: amount entered = [ ]

Observe: Did R enter amount correctly?

Clicked on help []

Debrief: How did you come up with your answer?

Included other sources: [] in rent fee [] in electricity bill OR [] no charge

Receive food stamps

Clicked on help []

Condo (y/n)

Condo fee (y/n)

Clicked on help [] at any point, describe when __________________________________

Debrief: Is the fee billed as a separate amount?

Monthly condo fee amount = ($ )

Observe: Did R enter amount correctly?

Tenure

Clicked on help []

Unit status (only in the path if no one is at the unit for 2 months or more) [] skipped question

Clicked on help []

If rented: Monthly rent= [$ ]

Observe: Did R enter amount correctly?

Include Meals Y/N

Clicked on help []

Property Value= [$ ]

Observe: Did R enter amount correctly?

Clicked on help []

(if owned) Taxes= [ $ ]

(if owned) Insurance = [$ ]

(if owned) Mortgage (y mortgage/ y contract to purchase/ n)= [ $ ]

Clicked on help []

(if owned) Second Mortgage (y home equity/ y second mortgage/ y both/ n)= [ $ ]

Clicked on help []

(Focus 1) Awareness or use of progress indicator

I’m going to pause you for a second:

About how much longer do you think the survey will take? How did you come up with your

answer?

(Focus 6) Let’s say you have to leave for an appointment, what would you do?

Observe: did they find “save & logout” button?

Observe: Use “save & logout” [] Close browser[] Other (explain)[]

(Focus 6) Let’s select “save & logout” right now. Pretend you have left for your appointment and have returned. You want to reenter the survey. What would you do?

Observe: if they mention PIN, and if they remember it.

Thanks for that information, okay you can continue

Person Questions

Place of Birth

Clicked on help []

Observe: Was there any difficulty entering the name of the place?

Did the R select the radio button first and then enter the name of the place? Use of drop down box or type ahead for state name?

Citizen & Year of entry into the U.S. (this can be 3 screens for those who are born out of U.S)

Clicked on help []

Schooling in last 3 months?

Clicked on help []

Highest level of schooling

Clicked on help []

Observe if try to check an answer category for each subheading:

Observe if R selected “Grade 1 through 11” and then specified grade. If not, what did they do after seeing edit message?

(if Bachelor’s degree or higher) Majors for Bachelors

Clicked on help []

Observe if there are issues entering majors in the field.

Ancestry

Clicked on help []

Since these examples are under the box, we may want to note if we think R is reading them.

Speaker of Other Languages

Clicked on help []

Observe: (asked in PRCS and also ACS)This is for PR so not sure how the question will be formulated.

Live here one year ago?

Clicked on help []

Observe: This is for PR (and ACS) so see where they put their address information.

Observe/debrief : what did R think checking the “Same address as John Doe” box would do?

Debrief: Are these the address categories you would use for a Puerto Rico address?

Health insurance or plans

Clicked on help []

Observe: See if they just check yeses or if they check no and yes, and if they can follow each category to the “yes” and “no”.

Q17: Deaf/blind

Clicked on help []

(ages 5+) Q18: Cognitive functions

Clicked on help []

(ages 15+) Q19: Can person go on errands

Clicked on help []

(ages 15+) Q20: Marital status

Clicked on help []

(ages 15+) Q21: 12 month marital status

Observe if R answered each question

Clicked on help []

(ages 15+) Q22: # times married

Clicked on help []

(ages 15+) Q23: year last married

Clicked on help []

(ages 15-50 and female) Q24: given birth

Clicked on help []

(ages 15-50) Q25a, b, c: grandchildren

Clicked on help []

(ages 15+) Q26 Served active military duty

Clicked on help []

(ages 15+) Q27 During which time period.

Observe and debrief: on each period to see if R checked all of the appropriate categories

Clicked on help []

(ages 15+) Q28 Disability rating

Clicked on help []

(ages 15+) Q29a Last week worked at job for pay (y/n)

Q29b: Last week worked as little as one hour for pay (y/n)

Clicked on help []

(ages 15+) Q30 Job address

Clicked on help []

Observe: See where the person enters the address information and what s/he does if s/he doesn’t know the information.

(ages 15+) Q31 Job transportation

Clicked on help []

(car, truck or van to Q31) Q32 # occupants

Clicked on help []

Q33: Time left for work ( )

Clicked on help []

Observe if they complete fields correctly, did they check am/pm radio button?

Q34: Commute time ( )

Clicked on help []

Observe if they complete fields correctly (only numbers)

(for the unemployed)

Q35a: Layoff y/n

Clicked on help []

Q35b: Temporarily absent last week y/n

Clicked on help []

Q35c: Return to work y/n

Clicked on help []

(for the unemployed) Q36: Looking for work y/n

Clicked on help []

(for the unemployed) Q37: Available for work y/n

Clicked on help []

(for the unemployed) Q38: Last work

Clicked on help []

(for the employed) Q39a: 50 weeks+

Clicked on help []

(for the employed) Q39b: #weeks

Clicked on help []

(for the employed) Q40: usual hours/week

Clicked on help []

(for the employed in past 5 years) Q41: type of employee (category= )

Clicked on help []

(for the employed in past 5 years) Q41: type of employee (category= )

Clicked on help []

Observe: did R try to choose more than 1 response?

(for the employed in past 5 years) Q42: employer

Clicked on help []

Observe: Was there enough space to write answer?

(for the employed in past 5 years) Q43: business or industry type 1

Clicked on help []

Observe: Was there enough space to write answer?

(for the employed in past 5 years) Q44: business or industry type 2

Clicked on help []

Observe: See if they tried to mark more than one

(for the employed in past 5 years) Q45: kind of work

Clicked on help []

Observe: Was there enough space to write answer?

(for the employed in past 5 years) Q46: activities or duties

Clicked on help []

Observe: Was there enough space to write answer?

Q47: income

Clicked on help [] , Which Category_________________________________

1. Wages [] no problem with either response category

[] problem with entering y/n , Describe__________________________________

[] problem with amount entry, Describe__________________________________

[] possible cognitive problem with item, probe during debriefing

2. Self-employment Income

[] no problem with either response category

[] problem with entering y/n , Describe__________________________________

[] problem with amount entry, Describe__________________________________

[] possible cognitive problem with item, probe during debriefing

3. Interest/dividends Income

[] no problem with either response category

[] problem with entering y/n , Describe__________________________________

[] problem with amount entry, Describe__________________________________

[] possible cognitive problem with item, probe during debriefing

4. Social Security

[] no problem with either response category

[] problem with entering y/n , Describe__________________________________

[] problem with amount entry, Describe__________________________________

[] possible cognitive problem with item, probe during debriefing

5. SSI

[] no problem with either response category

[] problem with entering y/n , Describe__________________________________

[] problem with amount entry, Describe__________________________________

[] possible cognitive problem with item, probe during debriefing

6. Public assistance/welfare

[] no problem with either response category

[] problem with entering y/n , Describe__________________________________

[] problem with amount entry, Describe__________________________________

[] possible cognitive problem with item, probe during debriefing

7. Retirement

[] no problem with either response category

[] problem with entering y/n , Describe__________________________________

[] problem with amount entry, Describe__________________________________

[] possible cognitive problem with item, probe during debriefing

8. Other income, examples VA/unemployment, child support, etc.

[] no problem with either response category

[] problem with entering y/n , Describe__________________________________

[] problem with amount entry, Describe__________________________________

[] possible cognitive problem with item, probe during debriefing

Observe:

(for the employed in past 5 years) Q48: Total income

Clicked on help []

Observe: Does the R notice the automatic sum (assuming they have programmed that way)

Does the respondent want to overwrite the sum?

Debrief: Does this sum make sense to you? What would you do if it didn’t seem correct?

Focus 2. Edit screens: So, in order to help alleviate some of the programming issues, we are getting rid of the screen about missing items, and going straight to the presummary screen which is where you seem to be now.

[] Edit Check 1

I’m going to pause you again:

Take a look at this screen, before making a selection on the screen, please tell me what you would do and why?

(if they chose submit) Why wouldn’t you select review first?

Now let’s go to review (anyway) and see how that screen works for you. This is the part where we probably want to direct them what to do on the Summary screen. Ask them how they would review the answers to their housing questions. Then how they would review for one person (give them a name on their roster). Then ask them to review a specific answer that would change their path and result in an additional question. So if they were born in the U.S. but now said they were born outside U.S. they would get different questions. Or if they said “No” to an income item but now say “yes” they would have to give us the amount. We can talk about how best to do this.

Did participant get lost in the instrument? [] Yes, explain below [] No

[] Edit Check 2

Observe: Did respondent select review first? [] Yes []No

(If selected review) Did participant get lost in the instrument? [] Yes, explain below [] No

General observations

Was there confusion on:

(Focus 7) Pick next person screen

Observe any problems: [] Yes, indicate what happened []No

(Focus 8) Observed respondent confusion with skipped screens

[] Yes, indicate what happened []No

(Focus 9) Unsolicited Participant comments on order of questions/topics:

People, Housing, People; Topic vs. Person

[] Yes, indicate what happened []No

Debrief: Do you have any comments on the order of the questions in this survey?

(Focus 10) Observed Navigational Problems (especially related to use of “previous/next”)

[] Yes, indicate what happened []No

(Focus 11) Observed participant confusion on name fills

[] Yes, indicate what happened []No

(PR focus)

Observed Translation issues [] Yes, explain below []No

Debrief: Do you have any comments on the Spanish translation used in the form? [] Yes []No

Unsolicited comments on color [] Yes ____________________ []No

Debrief: Do you have any comments on the color of the form? [] Yes []No

Unsolicited comments on banner image [] Yes ____________________ []No

Debrief: Do you have any comments on the banner at the top of the survey?

Unsolicited comments on length of labels/buttons [] Yes ____________________ []No

Debrief: Do you have any comments on the labels used on the response options or buttons?

Debriefing Questions

Pull up each screen and allow the participant to look at each screen as you ask questions about it. FOR Debriefing, you will need to log back in using the User ID and PIN and then enter this URL to get back to the beginning of the survey: http://idc4.ssd.census.gov:3122/acsx/pin

Debriefing:

Do you have any other comments on this survey?

Participant#:

RACE: White Black Hispanic Asian Other

AGE RANGE: < 30 31-45 46-60 61+

GENDER: M F

EDUCATION: HS/GED Some Coll/AA Bachelor’s Some grad

DATE OF INTERVIEW:

INTERVIEWER:

45 | Page