Various Pretesting Activities

1

Attachment A: ACS Internet Test Letters and Postcards

Where it applies, the bold condition titles and dates will be removed from the top of the
letters and postcards for the testing sessions. The participants will not see these titles. The
letters and postcards will be formatted to reflect the layout of the current ACS production
materials and the Administrative and Customer Services Division (ACSD) may also make
some design changes that will not affect the text of the letters or postcard.

1

1.1

Production

Figure 1. Production: Pre–Notice Letter

2

Figure 2. Production: Introductory Letter

3

Figure 3. Production: Reminder Postcard

4

Figure 4. Production: Replacement Letter

5

1.2

Not Prominent Internet Offer

Figure 5. Not Prominent: Pre-Notice Letter

6

Figure 6. Not Prominent: Introductory Letter

7

Figure 7.

Not Prominent: Reminder Postcard

8

Figure 8.

Not Prominent: Replacement Letter

9

1.3

Prominent Internet Offer (Choice)

Figure 9.

Prominent: Pre-Notice Letter

10

Figure 10. Prominent: Introductory Letter

11

Figure 11. Prominent: Reminder Postcard

12

Figure 12. Prominent: Replacement Letter

13

1.4

Internet Push on a Regular Schedule

Figure 13. Internet Push Regular Schedule: Pre-Notice Letter

14

Figure 14. Internet Push Regular Schedule: Introductory Letter

15

Figure 15. Internet Push Regular Schedule: Reminder Postcard

16

Figure 16. Internet Push Regular Schedule: Replacement Letter

17

1.5

Internet Push on a Modified Schedule

Figure 17. Internet Push Modified Schedule: Pre-Notice Letter

18

Figure 18. Internet Push Modified Schedule: Introductory Letter

19

Figure 19. Internet Push Modified Schedule: Replacement Letter

20

Figure 20. Internet Push Modified Schedule: Reminder Postcard

21

2

Attachment B: ACS Internet Screen Shots

These are the screen shots of the ACS Internet Instrument in their current form along with
the help text that is associated with them.
2.1

Welcome and Login Screens

Figure 21. Welcome Screen

22

Figure 22. Login Screen

23

2.2

Address Verification Screens

Figure 23. Address Verification

24

Figure 24. Address Verification 2

25

Figure 25. Address Verification 3

26

2.3

Roster Check

Figure 26. Roster Check

27

2.4

Sex

Figure 27. Sex

28

2.5

Age and Birthday

Figure 28. Age and Birthday

29

2.6

Hispanic Origin

Figure 29. Hispanic Origin

30

2.7

Race

Figure 30. Race

31

2.8

Type of Unit

Figure 31. Type of Unit

32

2.9

Property Value

Figure 32. Property Value

33

2.10

Place of Birth

Figure 33. Place of Birth

34

2.11

Highest Degree

Figure 34. Highest Degree

35

2.12

Ancestry/Ethnic Origin

Figure 35. Ancestry/Ethnic Origin

36

2.13

Language at Home/What Language?

Figure 36. Language at Home/What Language?

37

2.14

Health Insurance

Figure 37. Health Insurance

38

3

Attachment C: General Protocol

General Introduction
Please imagine that you have received each of these letters in the mail at home and read
them as you would normally read your mail.
Do you have any questions?
Start the eye-tracking software: Tobii Studio.
Hand them each letter in order for their assigned condition
3.1

Production Panel

Pre–Notice Letter
Imagine that you are opening your mail at home and that you received the letter that I am
going to show you now. Please read it as you would at home, and remember to think aloud
as you go through it. Give them the Pre–Notice Letter.
After they are finished reading the letter:
What do you think the letter is telling you? Anything else? Keeping probing with “Anything
else?” until the participant says “no.”
Did you think anything in the letter was confusing?
Introductory Letter
After a few days, you would receive this in the mail. This letter would be on top of the
American Community Survey questionnaire. Please read it as you would at home and
remember to think aloud as you go through it.
After they are finished reading the letter:
What do you think the letter is telling you? Anything else? Keeping probing with “Anything
else?” until the participant says “no.”
Based on this letter, how soon did you think that the Census Bureau expects you to complete
the survey?
Do you think that is a reasonable amount of time?
Did you think anything in the letter was confusing?
Reminder Postcard
After a few more days, you would receive this postcard in the mail. Please read it as you
would at home and remember to think aloud as you go through it.

39

After they are finished reading the postcard:
What do you think the postcard is telling you? Anything else? Keeping probing with “Anything else?” until the participant says “no.”
Based on this postcard, how soon did you think that the Census Bureau expects you to
complete the survey?
Do you think that is a reasonable amount of time?
Did you think anything in the postcard was confusing?
Replacement Letter
After a few weeks, you would receive this in the mail if you had not yet responded to the
survey. This letter would be on top of another copy of the American Community Survey
questionnaire. Please read it as you would at home and remember to think aloud as you go
through it.
After they are finished reading the letter:
What do you think the letter is telling you? Anything else? Keeping probing with “Anything
else?” until the participant says “no.”
Based on this letter, how soon did you think that the Census Bureau expects you to complete
the survey?
Do you think that is a reasonable amount of time?
Did you think anything in the letter was confusing?
How does the letter say you should complete the survey?
Was there any sentence in the letter that would encourage you to complete the survey?

3.2

Not Prominent Internet Offer Panel

Pre–Notice Letter
Imagine that you are opening your mail at home and that you received the letter that I am
going to show you now. Please read it as you would at home, and remember to think aloud
as you go through it. Give them the Pre–Notice Letter.
After they are finished reading the letter:
What do you think the letter is telling you? Anything else? Keeping probing with “Anything
else?” until the participant says “no.”
Did you think anything in the letter was confusing?

40

Introductory Letter
After a few days, you would receive this in the mail. This letter would be on top of the
American Community Survey questionnaire. Please read it as you would at home and
remember to think aloud as you go through it.
After they are finished reading the letter:
What do you think the letter is telling you? Anything else? Keeping probing with “Anything
else?” until the participant says “no.”
Based on this letter, how soon did you think that the Census Bureau expects you to complete
the survey?
Do you think that is a reasonable amount of time?
Did you think anything in the letter was confusing?
Reminder Postcard
After a few more days, you would receive this postcard in the mail. Please read it as you
would at home and remember to think aloud as you go through it.
After they are finished reading the postcard:
What do you think the postcard is telling you? Anything else? Keeping probing with “Anything else?” until the participant says “no.”
Based on this postcard, how soon did you think that the Census Bureau expects you to
complete the survey?
Do you think that is a reasonable amount of time?
Did you think anything in the postcard was confusing?
Replacement Letter
After a few weeks, you would receive this in the mail if you had not yet responded to the
survey. This letter would be on top of another copy of the American Community Survey
questionnaire. Please read it as you would at home and remember to think aloud as you go
through it.
After they are finished reading the letter:
What do you think the letter is telling you? Anything else? Keeping probing with “Anything
else?” until the participant says “no.”
Based on this letter, how soon did you think that the Census Bureau expects you to complete
the survey?

41

Do you think that is a reasonable amount of time?
Did you think anything in the letter was confusing?
Was there any sentence in the letter that would encourage you to complete the survey and
return it?
In your own words, what is a “personal visit?”
How would you feel about an interviewer conducting a personal visit at your home?
Would that sentence encourage you to complete the survey, or would it not affect you? Why
or why not?
• IF YES: On a scale of 1 to 9 with one being not at all and nine being very strongly,
how strongly would that sentence encourage you to complete the survey?
3.3

Prominent Internet Offer Panel

Pre–Notice Letter
A few days before the American Community Survey form comes in the mail, you would be
sent the letter that I am going to show you now. Please read it as you would at home, and
remember to think aloud as you go through it. Give them the Pre–Notice Letter.
After they are finished reading the letter:
What do you think the letter is telling you? Anything else? Keeping probing with “Anything
else?” until the participant says “no.”
Did you think anything in the letter was confusing?
Introductory Letter
After a few days, you would receive this in the mail. This letter would be on top of the
American Community Survey questionnaire. Please read it as you would at home and
remember to think aloud as you go through it.
After they are finished reading the letter:
What do you think the letter is telling you? Anything else? Keeping probing with “Anything
else?” until the participant says “no.”
Based on this letter, how soon did you think that the Census Bureau expects you to complete
the survey?
Do you think that is a reasonable amount of time?
Did you think anything in the letter was confusing?

42

How does this letter say you should complete the ACS? IF they say by the Internet,
probe: How do you feel about using the Internet to respond to the survey?
Reminder Postcard
After a few more days, you would receive this postcard in the mail. Please read it as you
would at home and remember to think aloud as you go through it.
After they are finished reading the postcard:
What do you think the postcard is telling you? Anything else? Keeping probing with “Anything else?” until the participant says “no.”
Based on this postcard, how soon did you think that the Census Bureau expects you to
complete the survey?
Do you think that is a reasonable amount of time?
Did you think anything in the letter was confusing?
How does this postcard say you should complete the survey?
Replacement Letter
After a few weeks, you would receive this in the mail if you had not yet responded to the
survey. This letter would be on top of another copy of the American Community Survey
questionnaire. Please read it as you would at home and remember to think aloud as you go
through it.
After they are finished reading the letter:
What do you think the letter is telling you? Anything else? Keeping probing with “Anything
else?” until the participant says “no.”
Based on this letter, how soon did you think that the Census Bureau expects you to complete
the survey?
Do you think that is a reasonable amount of time?
Did you think anything in the letter was confusing?
How does this letter say you should complete the ACS? IF They say by the Internet
here and did not say it for the Introductory Letter, probe: How do you feel about
using the Internet to respond to the survey?
Was there any sentence in the letter that would encourage you to complete the survey and
return it?
In your own words, what is a “personal visit?”
43

How would you feel about an interviewer conducting a personal visit at your home?
Would that sentence encourage you to complete the survey, or would it not affect you? Why
or why not?
• IF YES: On a scale of 1 to 9 with one being not at all and nine being very strongly,
how strongly would that sentence encourage you to complete the survey?
3.4

Internet Push Regular Schedule

Pre–Notice Letter
Imagine that you are opening your mail at home and that you received the letter that I am
going to show you now. Please read it as you would at home, and remember to think aloud
as you go through it. Give them the Pre–Notice Letter.
After they are finished reading the letter:
What do you think the letter is telling you? Anything else? Keeping probing with “Anything
else?” until the participant says “no.”
Did you think anything in the letter was confusing?
Introductory Letter
After a few days, you would receive this in the mail. Please read it as you would at home
and remember to think aloud as you go through it.
After they are finished reading the letter:
What do you think the letter is telling you? Anything else? Keeping probing with “Anything
else?” until the participant says “no.”
Based on this letter, how soon did you think that the Census Bureau expects you to complete
the survey?
Do you think that is a reasonable amount of time?
Did you think anything in the letter was confusing?
How does this letter say you should complete the ACS? IF They say by the Internet,
probe: How do you feel about using the Internet to respond to the survey?
Reminder Postcard
After a few more days, you would receive this postcard in the mail. Please read it as you
would at home and remember to think aloud as you go through it.
After they are finished reading the postcard:

44

What do you think the postcard is telling you? Anything else? Keeping probing with “Anything else?” until the participant says “no.”
Based on this postcard, how soon did you think that the Census Bureau expects you to
complete the survey?
Do you think that is a reasonable amount of time?
Did you think anything in the postcard was confusing?
How does this postcard say you should complete the ACS?
Replacement Letter
After a few weeks, you would receive this in the mail if you had not yet responded to
the survey. This letter would be on top of a copy of the American Community Survey
questionnaire. Please read it as you would at home and remember to think aloud as you go
through it.
After they are finished reading the letter:
What do you think the letter is telling you? Anything else? Keeping probing with “Anything
else?” until the participant says “no.”
Based on this letter, how soon did you think that the Census Bureau expects you to complete
the survey?
Do you think that is a reasonable amount of time?
Did you think anything in the letter was confusing?
How does this letter say you should complete the ACS?
IF They say by the Internet here and did not say it for the Introductory letter,
probe: How do you feel about using the Internet to respond to the survey?
In your own words, what is a “personal visit?”
How would you feel about an interviewer conducting a personal visit at your home?
Would that sentence strongly encourage you to complete the survey, or would it not affect
you? Why or why not?
• IF YES: On a scale of 1 to 9 with one being not at all and nine being very strongly,
how strongly would that sentence encourage you to complete the survey?
3.5

Internet Push Modified Schedule

Pre–Notice Letter
45

Imagine that you are opening your mail at home and that you received the letter that I am
going to show you now. Please read it as you would at home, and remember to think aloud
as you go through it. Give them the Pre–Notice Letter.
After they are finished reading the letter:
What do you think the letter is telling you? Anything else? Keeping probing with “Anything
else?” until the participant says “no.”
Did you think anything in the letter was confusing?
Introductory Letter
After a few days, you would receive this in the mail. Please read it as you would at home
and remember to think aloud as you go through it.
After they are finished reading the letter:
What do you think the letter is telling you? Anything else? Keeping probing with “Anything
else?” until the participant says “no.”
Based on this letter, how soon did you think that the Census Bureau expects you to complete
the survey?
Do you think that is a reasonable amount of time?
Did you think anything in the letter was confusing?
How does this letter say you should complete the ACS?
IF They say by the Internet, probe: How do you feel about using the Internet to
respond to the survey?
Replacement Letter
After a few weeks, you would receive this in the mail if you had not yet responded to
the survey. This letter would be on top of a copy of the American Community Survey
questionnaire. Please read it as you would at home and remember to think aloud as you go
through it.
After they are finished reading the letter:
What do you think the letter is telling you? Anything else? Keeping probing with “Anything
else?” until the participant says “no.”

46

How does this letter say you should complete the ACS?
IF They say by the Internet and did not say it for the Introductory Letter, probe:
How do you feel about using the Internet to respond to the survey?
Did you think anything in the letter was confusing?
Was there any sentence in the letter that would encourage you to complete the survey?
In your own words, what is a “personal visit?”
How would you feel about an interviewer conducting a personal visit at your home?
Would that sentence encourage you to complete the survey, or would it not affect you? Why
or why not?
• IF YES: On a scale of 1 to 9 with one being not at all and nine being very strongly,
how strongly would that sentence encourage you to complete the survey?
Reminder Postcard
After a few more weeks, you would receive this postcard in the mail. Please read it as you
would at home and remember to think aloud as you go through it.
After they are finished reading the postcard:
What do you think the postcard is telling you? Anything else? Keeping probing with “Anything else?” until the participant says “no.”
Based on this postcard, how soon did you think that the Census Bureau expects you to
complete the survey?
Do you think that is a reasonable amount of time?
Did you think anything in the postcard was confusing?
How does this postcard say you should complete the ACS?

Now that the session is complete, read this series of debriefing questions to the participant.
47

3.6

Debriefing Interview Questions

1. This question does not apply to the Production or Not Prominent conditions: Did you see the Internet response option on this letter/postcard? [Go through
each letter and postcard and ask if they saw it.]
2. This question only apploes to the Not Prominent and Prominent conditions:
Show them the paper questionnaire form. Did you see the Internet response option on
this form?
3. Who usually opens the mail in your household?
4. Who usually completes/fills out the survey forms or other forms for your household?
5. How often do you complete paper surveys and mail them back?
6. How often do you complete surveys on the Internet?
7. Do you prefer online surveys, paper surveys, or an option?
8. Would you complete a paper survey or questionnaire and return it?
9. Do you usually complete paper surveys and questionnaires immediately after opening
the mail, or do you wait until a later time to complete them?
• IF LATER: How long do you usually have the survey before you complete it?
• IF LATER: How much time typically passes between when you complete the
survey and when you mail it back?
• IF LATER: How many sittings does it take you to finish a survey completely?
10. Do you usually complete online surveys and questionnaires immediately after being
invited to take them, or do you wait until a later time to complete them?
• IF LATER: How long do you usually have the survey before you complete it?
• IF LATER: How many sittings does it take you to finish an online survey completely?
11. This question only applies to the Introductory letter for the Prominent Offer, Internet Push Regular Schedule, and Internet Push Modified Schedule
conditions. Give the Introductory letter back to the participant. Did you see the message about saving taxpayers money by responding to the ACS? Would that messsage
encourage you to complete the survey? Why or why not?
• IF YES: On a scale of 1 to 9 with one being not at all and nine being very strongly,
how strongly would that sentence encourage you to complete the survey?

48

12. This question only applies to the Introductory letter for the Prominent Offer, Internet Push Regular Schedule, and Internet Push Modified Schedule
conditions. Give the Introductory letter back to the participant. Did you see the
message about saving natural resources by responding to the ACS online? Would that
messsage encourage you to complete the survey? Why or why not?
• IF YES: On a scale of 1 to 9 with one being not at all and nine being very strongly,
how strongly would that sentence encourage you to complete the survey?
13. Other than what we have already talked about, did you have any other comments or
suggestions about the letters or postcard?

After the participant has finished with the test of the letters and postcard, bring them in to
testing Room A (5K512).
Hand them each letter packet for their assigned condition
Production: Pretend that you received this package in the mail at your home. Would
you open this package? IF YES: Please open the package and say a few words about your
impression of the materials inside. IF NO: Why not? / Please open the package and say a
few words about your impression of the materials inside.
Would you complete this survey form?
Would you prefer to take this survey online?
Why or why not?
Not Prominent Internet Offer: Pretend that you received this package in the mail at
your home. Would you open this package? IF YES: Please open the package and say a few
words about your impression of the materials inside. IF NO: Why not? / Please open the
package and say a few words about your impression of the materials inside.
Would you complete this survey online, by mail, or not at all?
Probe as appropriate about what they say they would do.
Prominent Internet Offer Pretend that you received this package in the mail at your
home. Would you open this package? IF YES: Please open the package and say a few words
about your impression of the materials inside. IF NO: Why not? / Please open the package
and say a few words about your impression of the materials inside.
Would you complete this survey online, by mail, or not at all?
Probe as appropriate about what they say they would do.
Internet Push Regular Schedule Pretend that you received this package in the mail at
your home. Would you open this package? IF YES: Please open the package and say a few
49

words about your impression of the materials inside. IF NO: Why not? / Please open the
package and say a few words about your impression of the materials inside.
Would you complete this survey?
Probe as appropriate about what they say they would do.
Internet Push Modified SchedulePretend that you received this package in the mail at
your home. Would you open this package? IF YES: Please open the package and say a few
words about your impression of the materials inside. IF NO: Why not? / Please open the
package and say a few words about your impression of the materials inside.
Would you complete this survey?
Probe as appropriate about what they say they would do.
Please complete the American Community Survey online for yourself only. I would like you
to tell me your impressions and thoughts about the Web survey as you take it. I would like
you to “think aloud” and talk to me about your impressions. If you expect to see some piece
of information, tell me about that expectation as well.
Finally, during the session, I may remind you to talk to me if you get quiet, not to interrupt
your thought process simply to remind you to talk to me. Please focus on verbalizing what
you are thinking as you read.

Now I am going to calibrate your eyes for the eye-tracking.
Do Calibration.
Now that we have your eyes calibrated, we are ready to begin. Please imagine that you have
received this mail packet in the mail. Please respond to the survey online as you would at
home. For the purposes of this study, please pretend that your name is Pat Smith and that
you speak French at home and live at the address on this envelope. Otherwise, you may
answer the questions as they apply to you in your real life.
Do you have any questions?
Start the eye-tracking software: Tobii Studio.
After the participant is finished, begin the debriefing questions.
Now that the session is complete, read this series of debriefing questions to the participant.
3.7

Debriefing Interview Questions
• What was your overall impression of the survey?

50

• Based on your experience with this online survey, would you be more likely to take this
survey online if you receive it, less likely to complete it, or would you say that your
likelihood of taking the survey was not affected?
• How does this survey compare to the Web surveys that you have taken in the past in
terms of visual appeal?
• How does this survey compare to the Web surveys that you have taken in the past in
terms of ease of use?
• Other than what we have already talked about, did you have any other comments or
suggestions about the Web survey?

51

4

Attachment D: Background Survey
Questionnaire on Demographics, Computer Use, and Internet Experience
YOUR ANSWERS ARE CONFIDENTIAL
Demographics

1. What is your age?
2. Are you male or female?
3. What is your level of education?
grade school
some high school
high school degree
some college
2–year college degree
4–year college degree
some postgraduate study (e.g., M.A., M.B.A., J.D., Ph.D., M.D., programs)
(e.g., M.A., M.B.A., J.D., Ph.D., M.D.)

Computer and Internet Experience
1. Where do you use a computer?
(Check all that apply.)
Home
Work
School
Library
Somewhere else- Please specify

52

2. If you use a computer in more than one location (for instance, both home and work),
which computer do you use most often?
(Check only one option.)
Home
Work
School
Library
Somewhere else
Please answer the following questions for the computer that you use most often
3. If you have a computer at home,
a. What kind of Internet access do you have at home?
Dial-up
Cable
DSL
Satellite
Cell Phone/3G/4G
Other
Don’t know

b. Which browser do you typically use at home? Please indicate the version if you can recall
it.
Firefox
Version
Internet Explorer
Version
Netscape
Version
Safari
Version
Other - Please specify
Don’t know

c. What operating system does your home computer run in? Please indicate the version if
you can recall it.
MAC OS
Version
Windows
Version
Linux
Version
Unix
Version
Other
Don’t know

53

4. On average, about how many hours do you spend on the Internet per day? Add up the
total hours spent on the Internet from all of the locations where you use it (for instance, at
home, at work, and at a library). Please do not count the time you spent checking email.
0 hours
1-3 hours
4-6 hours
7 or more hours

5. On average, about how many hours do you spend browsing Web sites on the Internet
using a cell phone?
0 hours
1-3 hours
4-6 hours
7 or more hours

6. Please rate your overall experience with the following (circle one number):

Computers

No Experience
1
2
3

4

5

6

Very Experienced
7
8
9

Internet

1

2

3

4

5

6

7

8

9

Social Networking

1

2

3

4

5

6

7

8

9

7. What computer applications do you use?
Mark (X) for all that apply
E-mail
Internet
Word processing (MS-Word, WordPerfect, etc.)
Spreadsheets (Excel, Lotus, Quattro, etc.)
Accounting or tax software
Engineering, scientific, or statistical software
Other applications, please specify

54

For the following questions, please circle one number.
Not Comfortable
8. How comfortable are you in learning to navi- 1
2
3
4
5
gate new Web sites?

6

7

Very Comfortable
8
9

9. Computer windows can minimized, resized,
and scrolled through. How comfortable are you
in manipulating a window?

1

2

3

4

5

6

7

8

9

10. How comfortable are you using, and navigating through the Internet?

1

2

3

4

5

6

7

8

9

Never
1
2

3

4

5

6

7

Very Often
8
9

12. How often do you perform complex analyses
of data using a computer?

1

2

3

4

5

6

7

8

9

13. How often do you use the Internet or Web
sites to find information? (e.g., printed reports,
news articles, data tables, blogs, etc.)
14. How often do you use social networking Web
sites like Facebook, Twitter, and LinkedIn?

1

2

3

4

5

6

7

8

9

1

2

3

4

5

6

7

8

9

Not Comfortable
1
2
3
4

5

6

7

8

1

11. How often do you work with any type of
data using a computer?

15. How comfortable are you with social networking Web sites like Facebook, Twitter, and
LinkedIn?
16. How comfortable are you with multitasking
while online. This could be chatting with someone through an instant messenger while surfing
through Web sites, or updating Facebook while
completing a survey.
17. How familiar are you with the Census
(terms, data, etc.)?
18. How familiar are you with the current American Community Survey (ACS) and American
FactFinder (AFF) sites (terms, data, etc.)?

55

2

3

4

5

6

7

8

Not Familiar
1
2
3

4

5

6

7

8

1

4

5

6

7

8

2

3

Very Comfortable
9

9

Very Familiar
9

9

5

Attachment E: Satisfaction Questionnaire

We will not do a separate satifaction questionnaire for the letters and postcard. The survey
will be in reference to the ACS Internet screens.
Please circle the numbers that most appropriately reflect your impressions about using the ACS
online survey.

1. What was your overall opinion about the Internet survey?

It was terrible
1
2
3
4

7

It was wonderful
8
9
Not applicable

7

Clear
8
9
Not applicable

7

Consistent
8
9
Not applicable

Less time than expected
1
2
3
4
5
6

7

More time than expected
8
9
Not applicable

Not Clear
1
2
3

7

Clear
8
9

Confusing
1
2
3

2. How clear were the instructions?

3. How consistent was the use of terminology
throughout the survey?

4. In your opinion, how long did it take you
complete the survey?

5. Was it clear how you were expected to complete the survey?

6. Additional Comments:

56

Inconsistent
1
2
3

4

4

4

5

5

5

5

6

6

6

6

Not applicable