2014 ACS Evaluation Report

November 12, 2014
2014 AMERICAN COMMUNITY SURVEY RESEARCH AND EVALUATION REPORT
MEMORANDUM SERIES #ACS 14-RER-29

MEMORANDUM FOR

ACS Research and Evaluation Advisory Group

From:

James Treat (signed on 11/19/2014)
Chief, American Community Survey Office

Prepared by:

Shelley Walker
American Community Survey Office

Subject:

American Community Survey: Online Visual Testing

Attached is the final American Community Survey Research and Evaluation report for the
American Community Survey Mail Package Research: Online Visual Testing. This study of US
adults who generally handle the mail for their households explored three alternative mail
package designs, as well as the current ACS mail package. The study had two goals 1) identify
ways to improve individual mail pieces (though elements such as design, layout, or messaging)
and 2) compare between the current ACS mail package (Control) and the three alternative mail
package designs (referred to as “Community,” “Official,” and “Patriotic” concepts) in order to
identify potential strengths and/or weaknesses. The report presents the findings from this
research.
If you have any questions about this report, please contact Shelley Walker on (301) 763-4045.
Attachment
cc:
ACS Research and Evaluation Work Group
ACSO PMGB
COM PMGB
ACS Integrated Communications Steering Committee
Ian O Brien
CNMP
Steven Jost
DIR
Jennifer Smits
PIO
Michael Cook

Messaging and Mail Package Assessment Research Subteam:
Agnes Kee
ACSO
Cheryl Chambers
Dameka Reese
Gary Chappell
Tasha Boone
Shelley Walker
Nancy Bates
ADRM
Laura Sewell
CNMP
Monica Wroblewski
Jennifer Hunter Childs
CSM
Eric Charles Newburger
DIR
Stephen Buckner
Jennifer Kim
DMD
Belkines Arenas Germosen
Thomas Chesnut
DSMD
Jennifer Guarino Tancreto DSSD
Anthony Tersine, Jr
Elizabeth Poehler
Justin Keller
EPCD
Timothy Olson
FLD
Michele Hedrick
PIO
Stacy Vidal
W Neil Tillman
Catherine Rosol
ERIMAX

American Community Survey
Mail Package Research:
Online Visual Testing

FINAL Report
Nov. 5, 2014

Sam Hagedorn, Michael Panek, Robert Green

Online Visual Testing Report

TABLE OF CONTENTS
Executive Summary

3

Project Background

4

Research Goals and Objectives

6

Methodology

8

Key Findings

20

Recommendations

46

Appendix A: Works Cited

48

Appendix B: In-Depth Findings

49

2

Online Visual Testing Report

EXECUTIVE SUMMARY
The American Community Survey Office (ACSO) is currently conducting a series of related research
projects around improving the American Community Survey (ACS) mail package and messaging toward
potential ACS respondents. This research aims to increase participation rates and reduce overall field
costs for the ACS by improving the effectiveness of the mail package.
This Online Visual Testing study of n=2,010 US adults who generally handle the mail for their households
explored three alternative mail package designs, as well as the current mail package. The Online Visual
Testing study had two goals: to identify ways to improve individual mail pieces (though elements such as
design, layout, or messaging) and to compare between the current mail package (Control) and the three
alternative mail package designs (referred to as “Community,” “Official,” and “Patriotic” concepts) in
order to identify potential strengths and/or weaknesses.
The study found opportunities to make the ACS mail package seem more eye-catching, important, and
authoritative to households. We present five key takeaways based on this study:


Visual design elements and deadline messaging can have a significant impact on how people
read and remember mail items. After looking at the Internet invitation envelopes, respondents
who saw the Official design were 2.8 times more likely to say the envelope was “urgent” than
those who saw the Control (58% “applies completely” vs. 33%, p<.01). They were also more
likely to agree it was “important” and “attention-grabbing.”



The U.S. Census Bureau logo should be prominently featured on mail items. Envelopes
containing the Census logo in the top left corner were more likely to be recognized as coming
from the Census Bureau and were more likely to be opened in the Mail Sort Exercise. As noted in
previous studies, the Census Bureau has significantly higher favorability than the Commerce
Department or the federal government as a whole.



Key words and phrases in letters should be emphasized using callout boxes, line spacing, and
bolded text. Elements like Web addresses, telephone numbers, and text that were enhanced
using graphic design techniques received more attention. For example, the Patriotic Internet
invitation used a blue accent box to call out the Web address to complete the survey: this item
was clicked earlier and more frequently in the Image Click Analyzer Exercise as compared to the
same content when featured less prominently in other mail designs.

3

Online Visual Testing Report


“Your response is required by law” attracts more attention than any other message. For
envelopes, letters, instruction cards, and reminder notices, the “mandatory” messaging clearly
caught participants’ attention. For example, nine in ten respondents highlighted the words
“required by law” in the Official pre-notice letter, which was more than three times greater than
the next most identified words. These test results strongly support continued use and further
experimentation to continually improve the mandatory message language on envelopes and
letters.



It is possible to overdo the commercial “marketing” look and feel. In the Mail Sort Exercise,
nearly one in three (31%) of respondents who initially saw the highly visual Lincoln Memorial
postcard indicated that they would place this item into the “Trash.” As the Census Bureau plans
field testing for alternative mail designs, we should be mindful of the tolerance threshold for
designs that might look like commercial marketing. On the other hand, designs that are formal yet
evocative may be a catalyst for drawing respondents into the ACS material. This is demonstrated
by the Patriotic design’s use of color, a flag motif, and American landmarks.

PROJECT BACKGROUND
The American Community Survey (ACS) collects detailed socioeconomic data from about 3.5 million
households in the United States and 36,000 households in Puerto Rico each year. The resulting
tabulations are provided publicly on an annual basis. ACS data are widely used inside and outside the
federal government, and play an important role in determining how more than $400 billion in federal
and state funds are distributed each year (Groves, 2012).
The ACS is a multi-modal survey. Households initially receive a series of mailings to encourage them to
respond online or by mail (see the 2009 ACS Design and Methodology Report for a full description).
These modes are identified as self-response. Then, Census Bureau representatives attempt to follow up
with the remaining households by telephone. Finally, in-person visits are made to a sub-sample of the
households that could not be contacted by telephone.
In 2012, just less than 60 percent of households self-responded to the ACS survey (Olson, 2013). A 2011
follow-up study indicates that the top two reasons given by non-response households for why they did
not respond to the ACS were: 1) they didn’t recall receiving mail about the ACS and 2) they didn’t open
the envelopes. This represents just over half (56%) of non-response households that received the “push”
mail strategy currently implemented by the ACS (Nichols, 2012).
These households represent a key area of opportunity for the ACS. Telephone and in-person ACS
completions are significantly more labor-intensive, and therefore expensive, for the Census Bureau. If
the Census Bureau could increase keeping and opening rates, it would see increased self-response rates,
lowering the costs associated with contacting non-responder households by phone and field contact. For
4

Online Visual Testing Report
example, the Census Bureau anticipates a net savings of more than $875,000 per year in nonresponse
follow-up costs by increasing the overall mail response rate by 1.6 percent after including an additional
reminder postcard (see Chesnut, 2010).
Previous studies have identified the importance of design in mail packages, and how it can account for a
significant amount of the variance in opening and read-through rates in direct mail campaigns (Feld et
al. 2013, De Wulf, Hoekstra, & Commandeur, 2000). It follows that by refining the design of the mail
packages, the ACS could increase opening and keeping rates, increase self-response rates, and reduce
costs.
The Census Bureau has previously conducted a variety of studies regarding the impact of the structure
and design of the ACS mail package on self-response rates. Most of these studies have focused on
adding or removing pieces from the mail package. For example, in 2010, the Census Bureau tested
through a repeated cognitive interview process different messaging and color on ACS letters and
envelopes that distinguished the ACS from the decennial enumeration (Schwede and Sorokin 2009).
Other tests have looked at the presence of icons compared with text-only instructions (Matthews et al.
2012), and different messaging approaches to reminder postcards (Schwede 2008).
The last time the Census Bureau conducted testing with a completely different visual design was in the
early 1990s. In that test, the more formal, “government” mail style dramatically outperformed the
more colorful, “marketing” approach—though the alternative design did not display that response was
required by law (Leslie, 1996).
The current ACS mail approach has developed through a series of Census Bureau research projects (see
Tancreto, 2012). The American Community Survey Office (ACSO) is currently conducting a series of
closely related research projects related to messaging and the ACS mail package with potential
respondents to the ACS. This research aims to increase participation rates in the ACS survey and reduce
the amount of follow-up activities with non-response households.

Current ACS Mail Package

5

Online Visual Testing Report
This study is the next step in a series of projects Team Reingold has taken towards developing more
effective ACS mail packages. In early 2014, the contractor team conducted several studies to develop
effective messaging strategies that could be used in ACS mail pieces to increase self-response rates.
These projects included Mental Modeling based on n=25 field interviews; n=7 Deliberative Focus
Groups; a two-wave Messaging Survey of n=2,015 telephone respondents; and in-depth Key Informant
Interviews with n=109 ACS and community stakeholders. Together, these projects identified top
messaging themes that are most likely to increase response rates among those who receive ACS pieces
in the mail.
Using these results, the contractor team developed three alternative mail package designs and revised
them through a series of n=6 focus groups and n=34 one-on-one interviews. We also incorporated
recommendations from an independent review by expert mail-out survey researchers (Dillman, 2014).
This study uses online visual testing to refine and improve the proposed designs of these three
alternative ACS mail packages. After the findings and recommendations from this Online Visual Testing
have been incorporated into design revisions, the ACSO will design and implement a field test based on
the designs. The goal of this test will be to measure real-world changes in response rates and monitor
potential data quality issues. This may include testing variations on the designs to identify whether
specific elements of the package improve or harm response rates.

RESEARCH GOALS AND OBJECTIVES
The Online Visual Testing study had two goals: to identify ways to improve individual mail pieces
(though elements such as design, layout, or messaging) and to compare between the current mail
package (Control) and the three alternative mail package designs in order to identify potential strengths
and/or weaknesses. These findings can be used to inform a subsequent real-world mail test.

The survey used four designs — each consisting of several mail pieces thematically and visually linked to
each other. This included the three contractor team-developed alternative mail packages (“Official,”
“Community,” and “Patriotic”), as well as a control condition of the current mail package.

6

Online Visual Testing Report

Four Mail Package Designs
“Official”

“Community”

“Patriotic”

“Control”

The Online Visual Testing design was monadic: any given respondent saw just one of the four designs.
Each of the four test cells had roughly identical demographic characteristics (gender, age, and
race/ethnicity). Respondents went through a series of online exercises that follow the progression of
mail items in the mail package (described in more detail in the Methodology section).
In the online survey, we tested many — though not all — of the items in the mail package. Some
elements of the mail package that do not vary between the designs were not tested, including the
return envelopes that are part of the paper questionnaire mailing. Likewise, the actual paper
questionnaire for the ACS is beyond the scope of this project and was not tested.
The findings of this study are designed to provide guidance for internal Census Bureau decision-making
only. They are not intended for publication or public dissemination as official estimates. While the
results may inform ACS messaging and subsequent research, results will not drive significant policy
decisions.
Please note in this report the terms “respondent” and “participant” are used interchangeably to refer to
the U.S. adults who generally handle the mail for their household who completed interviews for this
study.

7

Online Visual Testing Report

METHODOLOGY
Universe and Respondent Selection
In order to measure the perceptions and reactions of those that are most likely to interact with the ACS
mail package, the universe for this study was U.S. adults (18 years or older) who generally handle the
mail for their household.
According to ACS estimates, there were approximately 131 million households in the United States in
2012 (Olson, 2013). As reliable demographic estimates of the population who handles the mail were not
available, we used householder (head-of-household) data from the Current Population Survey (CPS) as a
proxy for the sample probability weights and quotas. While this is not a perfect proxy for the target
universe, it provided a reasonable framework to represent the adults who generally handle the mail for
their household.
Respondents were selected from the GfK Knowledge Network (KN) online panel, which consists of
approximately 50,000 adult members in the United States. Randomly sampled addresses from the
United States Postal Service Delivery Sequence File are invited to join the panel through a 12-week
recruitment process. Initially, households are sent a recruitment packet with a $2 cash incentive.
Approximately one week later, a reminder postcard is sent to all addresses. After three weeks, a letter is
set to households that have not responded. After five weeks, KN makes recruitment calls to nonresponding households that can be matched to a telephone number (approximately 70% of remaining
addresses). The response rate for the ACS mail recruitment is approximately 14% (AAPOR 3). As part of
compensation for their participation in the panel, respondents without Internet access receive a
netbook and Internet access from the online panel vendor.
Before participants are eligible to become active members of the panel, they must complete profiling
questions for demographic and household characteristics including age, race, and gender.
Approximately 72% of “recruited” persons complete the profile questions and become active members
of the panel. On average, most panel members participate in about two surveys per month.
For this study, Knowledge Networks sent out a total of 8,186 email invitations for this study. Of these,
4,480 respondents completed the screening questions, yielding a screening completion rate (S.COMP) of
54.7%. Of those, n=2,887 qualified for the survey by typically handling the mail for their household (70%
eligibility rate). A total of n= 2,009 respondents completed the survey, yielding a final cumulative
response rate of 5.5% (CUMRR1: 14% panel recruitment x 72% profiling rate x 55% cooperation rate, see
Callegaro & Disorga 2008). Note that partial interviews are not included in this total.
As a result, it is possible that the mail-handling US adults that completed our survey may differ from the
population at large, particularity around their propensity to respond to surveys. When interpreting the
study’s results, it may be useful to be mindful that the findings from this study could differ from the
population at large. Subsequent real-world field testing is necessary to confirm the findings of this study
before making permanent changes to the ASCS mail package.
8

Online Visual Testing Report

Sampling Quotas
While the survey will not be used to make estimates of the target population as a whole, the sample
used a selection weight and demographic quotas to ensure that the findings were not unduly influenced
by sample imbalances between mail packages in demographic characteristics such as race, age, and
gender. In the survey, we screened for an adult who generally handles the mail for their household.
We created four sub-samples (one for each of the four mail package treatment conditions) with roughly
equal demographic characteristics in terms of age, race/ethnicity, and gender. Participants were
randomly assigned into one of the four packages (“Official,” “Community,” “Patriotic,” “Control”). If a
participant with particular demographic characteristics was placed into a quota that was full, he or she
was randomly coded into a package that was still open. Once all four packages were full for a particular
set of demographic characteristics, no more cases with those characteristics were eligible for the survey.
The sample for the online visual testing was drawn using a probability proportional to size (PPS)
weighted sampling approach based on head-of-household data from the Current Population Survey
(CPS). We used householder (head-of-household) data from the Current Population Survey (CPS) as a
proxy for the sample weights and demographic quotas since reliable estimates of the portion of the U.S.
population who handles the mail are not available.
There was one notable adjustment between CPS householder data and the demographic quotas. Gender
targets in the online visual testing survey were constructed by combining the number of householders
living in non-family households and family households where no spouse is present for each gender. In
addition, married couples living in the same household were considered equally likely to have a male or
a female who handles the mail, so as not to over-represent the proportion of married families that
identify the male as the householder for the family. This is summarized in the following table:
Number of Householders by gender and family status (in thousands)
Householder Family Status
Total
Male
Householder not in family household
(includes living alone or with nonrelatives)
Householder in family without spouse
(includes married spouse absent, widowed, divorced, separated, or
never married)
Married with spouse present
(* for weighting purposes, married spouses in the same household
are considered equally likely to generally handle the mail)
Total Householders
Percentage
(Source: CPS 2013 Annual Social and Economic Supplement, Tables H2 and A2)

9

Female

41,558

19,747

21,810

21,699

6,230

15,469

59,204

29,602*

29,602*

122,460
100%

55,579
45%

66,881
55%

Online Visual Testing Report
We used quotas to balance age, gender, and racial/ethnic characteristics. The quotas used a crossdesign (age x gender x racial/ethnic group; for the full quota table see Appendix B: In-depth Findings).
The following table provides an overview of the demographic quotas for the survey:

Category
Gender

Race

Age

Demographic Targets for Quotas
Characteristic
Target Percentage
Male
45%
Female
55%
White, non-Hispanic
72%
Hispanic, any race
11%
Black, non-Hispanic
12%
Asian, non-Hispanic
4%
All Others/multiracial
1%
18-29
13%
30-44
26%
45-64
38%
65+
23%

(Source: CPS 2013 Annual Social and Economic Supplement, Tables H2, H3, and A2)

Procedures for Collecting Information
Respondents self-administered the survey through an online portal on their computers. As part of the
survey, participants completed a series of exercises that tested various pieces of the ACS mail package in
the general order the items are received (starting with the pre-notice envelope). Participants did not
see the names of the design they evaluated.
Four Mail Package Designs
“Official”

“Community”

“Patriotic”

“Control”

10

Online Visual Testing Report

Overview of Activities
First, respondents saw a short series of screening questions to ensure they qualified for the survey.
Next, they participated in three activities with the objective of assessing different aspects of mail
package designs:
1. Mail Stack Exercise: This exercise presented participants with an interactive simulation of mail
sorting and asked them to sort a collection of mail, including the pre-notice envelope, the
Internet invitation envelope, or the reminder postcard for the particular mail package design
they were assessing. There were also six pieces of non-Census Bureau mail. Respondents
participated in the mail stack exercise once.
2. Image Click Analyzer: Respondents were asked to click on particular elements of the designs in
order to illustrate what attracted their attention, to determine how participants interact with
the design, and to understand what drives engagement. Respondents used image click analyzer
tools a total of four times on several pieces of mail.
3. Message Highlighting: This section asked respondents to highlight words and phrases that they
found compelling, first for the text of the pre-notice letter (Pre2) and then for the Internet
response letter (A3).
For each of the tools used as part of the online visual testing study, we describe in this section the
functions of the tool, the user experience, the data variables that the survey instrument collected, and
how the research team analyzed the data and developed recommendations. Note that participants
were not required to answer any particular question, though only a small portion (less than 5% of
responses per question) are missing from the data set. For analysis purposes, missing responses were
excluded from analysis for that particular question.
Current Mail Package Design

11

Online Visual Testing Report
Proposed Alternative Mail Package Design

Exercise

Mail Pieces Tested

# of sides

Two ACS Mail pieces - pre-notice envelope (pre1) OR
2
Mail stack exercise (#1)
Envelope (A1/C1) OR reminder postcard (D1) AND six
(front/back)
pieces of “clutter” representing other mail items
Image click analyzer (#1) Pre-notice letter (Pre2)
1
Message highlighting (#1) Pre-notice letter (Pre2)
1
Image click analyzer (#2) Envelope (A1/C1) (either front or back)
2
Image click analyzer (#3) Internet invitation letter (A3)
1
Message highlighting (#2) Internet invitation letter (A3)
1
Instruction card (A2) OR Instruction card: choice (C2) OR
2
Image click analyzer (#4)
reminder postcard (D1)
(front/back)

n-size
(per design/side)

333
500
500
250
500
500
125

Mail Stack Exercise
The Mail Stack Exercise simulated receiving a selection of print mail to assess whether designs were
noticed, opened, and recalled. This study included a single mail sort exercise at the start of the survey
exercises to minimize the risk of priming effects when the participant has already seen ACS mail items
earlier in the survey. By simulating real-world letter-sorting, we were able to determine which envelope
designs are more likely to help ACS items get opened among mail-handling adults. We also used followup questions to provide insights into recall and retention of the designs. Previous studies have identified
the importance of design in mail packages, and how it can account for a significant amount of the
variance in opening and keeping rates in direct mail campaigns (Feld et al. 2013, De Wulf, Hoekstra, &
Commandeur, 2000).

User Experience
During the Mail Stack Exercise, respondents saw a total of eight mail pieces in a random order. Two of
the mail pieces were part of the ACS (Pre1 and A1/C1). While in the current ACS addresses receive a
single mailing at a time, we included two ACS items in the Mail Sort to allow us to test a wider range of
materials. The rest of the mail was part of a “clutter reel” that is representative of the most common
12

Online Visual Testing Report
types of mail an average household receives, such as product offers, promotions, letters, and bills (see
“clutter reel” section). The order of the mail was randomized for all pieces to control for order bias.
Participants viewed the mail pieces on their screen and flipped over the mail item by clicking the button
at the top of the screen (revealing the reverse-side). Participants were able to click on a design and sort
it into either to “save and read” or send to “trash.”

The Mail Stack Exercise used a “click-and-drag” system to simulate letter sorting.

After the Mail Stack Exercise, respondents saw a series of follow-up questions to provide insights
surrounding their recall and retention of ACS packaging. First, participants were asked an open-ended
question concerning recall (unaided) about the mail items they just reviewed. For the second question,
half of participants were probed as to whether or not they saw anything from the federal government
(with an open-ended follow-up if yes). The other half was asked if they saw anything from the Census
Bureau (with an open-ended follow-up if yes).

“Clutter reel”
The average household receives 19 pieces of mail each week (Mazzone & Rehman, 2012). This study
used six mock non-ACS mail pieces which allowed us to test ACS mail packages against personal mail,
transactional mail (such as bills and financial statements), and advertisements, without dramatically
increasing respondent fatigue.
Household Mail Volume Received and Sent by Market Served (2012)
Mail by Market Type

Number of Pieces (in billions)

Correspondence
Transactions (includes bills, statements)
Advertising
Periodicals (includes USPS-delivered
newspapers and magazines)
Other (includes packages and
13

Percentage

11.8
34.3
79.6

8.6%
24.9%
57.8%

5.1

3.7%

6.9

5.0%

Online Visual Testing Report
unclassifiable)
Total

137.7

100%

Source: Mazzone & Rehman (2012) USPS Household Diary Survey, Table E-2.

The most common senders of advertising mail are merchants (34% of advertising), financial (26%),
services (19%), and social (18%) (ibid., Figure 5.2). The most frequently sent merchant mail is from a
department store or automotive dealer. The most frequently sent first-class financial mail is from
insurance companies. The most frequent service is a utility provider (ibid. Figure A2-8).
As such, we included the following mock items in the “clutter reel”:
 3 advertisements (insurance, department store, and automotive dealer)
 2 bills (electrical and water)
 1 personal letter envelope (hand-addressed)

Data collected
The Mail Stack Exercise was used to gather rich data from hundreds of respondents’ interaction with the
ACS mail package in order to evaluate and refine proposed designs for use in the ACS field test.
Specifically, the Mail Stack Exercise collected four data points:






Which side of the mail package the participant saw first (either the front or the back,
randomized for each individual item in the mail stack)
Whether the participant flipped the mail item over (either “did’ or “did not”)
How long the participant viewed each side of the mail item (measured in half-seconds
from when the initial image loads on the participants screen until they make a
decision)
What decision the participant made (either “save and read” or “trash”)

It is worth noting that the Mail Sort Exercise was not designed to directly measure real-world letter
opening rates, but to provide useful guidance into respondents’ intent to open envelopes based on their
visual design. While field testing the mail packages will provide more robust data regarding the impact
on real-world response rates, using online experiments to improve the design of envelopes and other
mail elements allows the Census Bureau to acquire more information about what drives decisions to
open, keep, and respond to the ACS invitation.

Analysis and recommendations
We examined differences between mail packages in terms of respondents’ decisions. This analysis was
conducted for the front and back of the envelopes separately, allowing us to identify if a particular side
of an envelope yielded lower results.
There were a total of ten designs between the pre-notice envelope, the Internet Invitation envelope
(A1/C1) and the reminder postcard (B1). Given that each item had two sides, we analyzed 20 sides of
mail in total.
14

Online Visual Testing Report

There were four possible outcomes for each side of the envelope: select “save and read” without
flipping over the envelope, select “trash” without flipping, “save and read” after flipping, and “trash”
after flipping the envelope. For each of the ten envelope sides, we measured the overall proportion of
outcomes to assess if there are sides that underperform, and recommend changes to improve the
designs. In general, we posit that mail designs that are quickly flagged as worth “saving” are more
effective than items that take more time to process when it comes to decision making.

Image Click Analyzer
User experience
The Image Click Analyzer explored how participants interact with particular designs. Respondents used
the Image Click Analyzer a total of four times (Pre2, A1/C1, A3, and either A2, C2, or D1). The exercise
asked respondents to click on the areas of designs that caught their attention, illustrating the visual
hierarchy of various layouts by mapping the order of different sections and how the flow of the design
works in a quantitative way. It also identified where respondents’ eyes were drawn to when they looked
at visual elements, such as logos, headings, or bolded words.

Image Click Analyzer asks participants to click on parts of the image that draw their attention.
After clicking on the image, participants were able to explore the mail item in an unstructured way,
including being able to flip the mail item over. Participants then answered two exploratory open-ended
questions — what they recalled from the document, and whether there were any parts that were
unclear or confusing. These questions were then coded by two members of the research team who had
not previously been engaged in the project, and the resulting codes were reviewed by a third member of
the research team.

15

Online Visual Testing Report
After the envelope (A1/C1) was tested, we asked participants how strongly they agreed with a range of
adjectives applying to the design, including “engaging,” “official,” “trustworthy,” and “urgent.”

Data collected
This tool collected location information based on where the participant clicked. The information was
encoded as a coordinate system. In addition, the coordinates were sub-divided into design elements
that helped organize data analysis.
Click-testing Image Layers

Visible Layer

Coordinate Grid

Design Elements

Analysis and recommendations
As each mail piece had different elements and visual arrangements, image click data were analyzed for
individual sides of the design, rather than comparing between sides.
During analysis, we used the data to generate a “heat map” showing the areas that respondents clicked
most frequently. The heat map portrayed point-data representing individual clicks. We also reported
“click frequency tables” which display the percentage of respondents that clicked on each design
element in the piece. Heat maps and click frequency tables for each piece can be found in Appendix B:
In-depth Findings.
Example “click frequency table” from Image Click Analyzer
(%)

Click
st

1 Click

2 Click

3rd Click

Logo

70%

25%

26%

Stamp

0%

1%

0%

Return 1

<1%

2%

10%

Return 2

<1%

1%

1%

Address

5%

9%

26%

Image (Virginia)

4%

6%

9%

Required

19%

54%

23%

Footer

1%

1%

4%

16

nd

Online Visual Testing Report

Example “heat maps” from Image Click Analyzer

1st Click

2nd Click

3rd Click

17

Online Visual Testing Report

Message Highlighting
The Message Highlight tool focused on words, phrases, and messages within the text, rather than the
visual design elements. We used this tool to assess the pre-notice letter (Pre2) and Internet response
letter (A3) to help ensure that the alternative designs convey the most important pieces of information
from each letter.

User experience
Respondents saw the text of the letter on screen in a selectable format. Using their mouse, respondents
highlighted words that they found most compelling. The words they selected did not have to be
continuous, and there was a button that allowed them to clear the selection if they wanted to change
the text they highlighted.

An example of Message Highlighting
After completing the highlights, respondents were asked one of two open-ended questions (n=250
responses per design for each question): “If you had to explain to a friend the purpose of the letter you
just read, how would you describe it to them?” or “Thinking about the letter you just read, were there
any parts you found to be unclear or confusing?”

Data collected
In the message highlighting, the survey collected the frequency of which words were highlighted when
participants submitted their answer. We also collected the open-ended responses as text variables.

18

Online Visual Testing Report

Data Analysis
We conducted analysis for a total of eight letters (including both the pre-notice and Internet invitation
letters for all four designs).
The research team created a visual representation of which words get highlighted frequently by having
the frequency of selection represented in the size and color of the word. Words that were selected more
often are physically larger than less selected words. They are also colored red, whereas average selected
words are colored in purple and less selected words are colored in grey. A visual representation for each
letter we tested can be found in Appendix B: In-depth Research Report.
Example Message highlighting visual representation

We also created word-by-word frequency charts, which report the percentage of respondents that
highlighted each word in the letter. Word-by-word frequency charts for each letter we tested can be
found in Appendix B: In-depth Findings.
The open-ended questions were coded into bins to identify general themes in free-response answers.
After an initial codebook was developed through an initial review of the responses, the questions were
coded by two members of the research team who had not been previously engaged in the project, and
the coded cases were reviewed by a third member of the research team. Tables with coded freeresponse answers for each piece we tested can be found in Appendix B: In-Depth Findings.
In the analysis, we noted where key information might be overlooked (for example, what the participant
is being asked to do), or where particular words and phrases tended to attract more attention. In
addition, particularly insightful verbatim quotes from the open-ended responses were included in the
report.

19

Online Visual Testing Report

Closing questions
The final set of questions was intended to gauge participants’ perceptions of the range of pieces that
they saw during the survey. Findings from these closing questions helped identify the “tone” of
particular packages as a whole by assessing how well respondents felt certain descriptive terms applied
to the particular design they reviewed.
We also asked participants to assess how likely they would be to participate in the ACS if they received
these mailings at their home, and what they perceived the purpose of the ACS to be. By comparing
between packages, we were able to assess if some of the different thematic elements employed by
particular designs were more effective at communicating the purpose of the ACS and the value of
participation.
As this study was conducted under the CLMSO’s Generic Clearance for Data User and Customer
Evaluation, this study should not be used to draw inferences regarding the country’s population at-large
and should not be used to publish any official statistical estimates.

KEY FINDINGS
The goal of the Online Visual Testing study is to identify the strengths and weaknesses of each
alternative mail package (Official, Community, and Patriotic) as well as the Control package. In addition,
we conducted an analysis to identify recommendations for revising individual mail pieces.
For the analysis, we used a variety of different exercises and analytical techniques. Appendix B: In-Depth
Findings presents the full findings, including crosstabs for each mail package. Our analysis includes:






Mail Sort Exercise
Coding open-ended responses to assess unaided and aided recall
Image Click Analyzing
Messaging Highlighting
Attitudinal questions

Mail Sort Exercise
Differences between types of mail
In the Mail Sort Exercise, participants were presented with a stack of eight mail items. Two of the mail
items were from the Census Bureau (the pre-notice envelope, Internet invitation envelope, or reminder
postcard). The remaining six items represented a broad variety of mail that typical households receive
including advertisements, bills, and personal letters. Participants sorted the mail items into items they
would “Save and Read Later” and items they would send to the “Trash.”
As illustrated in the table below, adults who handle the mail for their household identified Census
materials, bills, and personal letters as items they would be interested to keep and read at a later point.
Most of them discarded advertising materials at a much higher rate, with four in five “trashing”
20

Online Visual Testing Report
department store advertisements and an even higher rate ”trashing” the dental and automotive
advertisements.
Mail Item

% overall selected “Save and Read Later”

% overall selected “Trash”

Census Internet invitation envelope*
91%
9%
Water bill
86%
14%
Personal, hand-written card
85%
15%
Census reminder postcard*
85%
15%
Census pre-notice envelope*
84%
16%
EDA bill
70%
30%
Department store advertisement
20%
80%
Dental insurance advertisement
13%
87%
Car advertisement
5%
95%
*Average of four alternative design packages (Official, Community, Patriotic, and Control).

Differences between pre-notice mail designs
For the pre-notice letter, we tested four envelopes with different variations of logos and placement of
standard-language text (such as a mail ID code, “OFFICIAL BUSINESS,” or “An Equal Opportunity
Employer”). The back sides of the envelopes were blank.
We saw a significant difference in participants’ save/trash rates depending on whether the participant
began by seeing the front or the back of the envelope. 91% of participants who began by seeing the
front of one envelope selected “Save and Read Later,” compared with 79% of participants who saw the
back first (p<.01). In part, this is because a third (33%) of people who began seeing the blank back of the
envelope did not flip over the envelope to see the front side. This could be a mode effect from the
online testing, so for analysis purposes we focused specifically on the individuals who first viewed the
front side of the envelope.
We found that the pre-notice envelopes with the Census Bureau logo in the top-left corner had a
statistically significantly higher proportion select “Save and Read Later” than the Control with a text-only
“U.S. DEPARTMENT OF COMMERCE” header at the a=.05 level.

21

Online Visual Testing Report
Alternative Designs

Control

Alternative Designs: 92%

Control: 87%

select “Save and Read”

select “Save and Read” (p<.05)

Control

There was not a statistically significant difference between the three alternative pre-notice letter
designs in terms of “Save and Read Later” (p>.40). This suggests that the Census Bureau logo is the
driving factor for the differences observed between the alternative mail packages and the control. This
reinforces previous survey findings that the Census Bureau is an exceptionally well-known and respected
organization to the public (Hagedorn & Green, 2014).
Official

Community

Patriotic

Official: 93% select “Save and Read”

Community: 91% select “Save and Read”

Patriotic: 93% select “Save and Read”

Differences between Internet invitation mail designs
In the mail sort exercise, we also tested four different envelopes for the Internet and mail invitations.
These envelopes contain the materials to respond to the ACS (instruction cards, paper questionnaires,
FAQs). Participants initially saw either the front or back of the envelope. For the three alternative mail
pieces, only a small portion of respondents flipped over the envelope from the side they initially saw.
22

Online Visual Testing Report
Roughly one in four (26%) respondents who started looking at the back of the Community Internet
invitation flipped it over. This is likely a mode effect from the survey instrument, rather than a realworld behavior. As a result, we have focused our analysis on the initial side that respondents saw first.
The following table shows the percentage that selected “Save and Read Later” for each side of the
envelope separately.

Official

Mail Sort: Internet Invitation Mail Sort (A1/C1)

95%

10%

selected
“Save and Read Later”

flipped over mail item to view
back side

Among those who saw the back first:

F

Community

% “Save and Read Later”
% Flip Over
by Initial Side
Among those who saw the front first:

B

86%

18%

selected
“Save and Read Later”

flipped over mail item to view
front side

Among those who saw the front first:

90%

10%

selected
“Save and Read Later”

flipped over mail item to view
back side

Among those who saw the back first:

F

B

96%

26%

selected
“Save and Read Later”

flipped over mail item to view
front side

Patriotic

Among those who saw the front first:

94%

8%

selected
“Save and Read Later”

flipped over mail item to view
back side

Among those who saw the back first:

F

B

94%

18%

selected
“Save and Read Later”

flipped over mail item to view
front side

Control

Among those who saw the front first:

92%

18%

selected
“Save and Read Later”

flipped over mail item to view
back side

Among those who saw the back first:

F

B

78%

67%

selected
“Save and Read Later”

flipped over mail item to view
front side

On the Official envelope, we saw a significant difference between respondents who saw the front of the
letter first (95% selected “Save and Read Later”) and those who saw the back of the letter first (86%,
p<.01). We explore whether or not the visual hierarchy and the positioning of the US Census Bureau
logo on the top right corner of the envelope contributed to this in the Click Image Analyzer section.
The Community envelope had the least effective front (90% selected “Save and Read Later”) and the
most effective back (96% selected “Save and Read Later”). The back of this envelope, which featured a
vertical stacked design and accent colors, had the highest percentage of participants choose to read,
though the difference is not statistically significant from the back of the Patriotic envelope.
The Patriotic envelope, which featured a multi-colored footer of US landmarks and a flag motif, had 94%
of participants select “Save and Read Later” – regardless of which side they saw first. The similar
response rates between those who initially viewed the front first or the back first suggests that the

23

Online Visual Testing Report
footer elements employed on both sides of the designs may receive more attention from participants
than the other elements.
Among those who initially saw the front of the Control, 91% of people selected to “Save and Read Later”
the envelope. We find the fact that only two thirds of participants flipped the blank back of the
envelope over to view the front side a less credible indicator.

Differences between reminder postcards
In the Mail Sort Exercise, each of the mail designs had a corresponding reminder postcard. These postcards
use a range of visual designs from a text-heavy approach (Control) to a more graphic-heavy approach
(Patriotic). The following table displays the proportions of participants who selected “Save and Read Later”
by which side the participant saw first, as well as the proportions that flipped over the card.
% “Save and Read Later”
by Initial Side

Mail Sort: Reminder Postcard (B1)

% Flip Over

Among those who saw the front first:

Official

93%

84%
F

17%

selected
flipped over mail item to view
“Save and Read Later”
front side
Among those who saw the front first:

B

90%

13%

selected
flipped over mail item to view
“Save and Read Later”
back side
Among those who saw the back first:

79%
F

11%

selected
flipped over mail item to view
“Save and Read Later”
front side
Among those who saw the front first:

B

91%

Patriotic

Community

12%

selected
flipped over mail item to view
“Save and Read Later”
back side
Among those who saw the back first:

9%

selected
flipped over mail item to view
“Save and Read Later”
back side
Among those who saw the back first:

69%
F

14%

selected
flipped over mail item to view
“Save and Read Later”
front side
Among those who saw the front first

B

Control

91%

19%

selected
flipped over mail item to view
“Save and Read Later”
back side
Among those who saw the back first:

F

B

77%

10%

selected
“Save and Read Later”

flipped over mail item to view
front side

The most striking finding is the 22 percentage point gap between the Patriotic front (91% selected “Save
and Read Later”) and the Patriotic back (69% selected “Save and Read Later”). Nearly one in three (31%)
respondents who initially saw the card that featured the Lincoln Memorial discarded it in the “Trash,”
significantly higher than the proportions for backs of other reminder postcards (p<.01). This suggests
that upon first glance, this particular design does not appear “formal” enough to be recognized as a
legitimate government notification or communication.
24

Online Visual Testing Report

The Official design had the highest proportion for both the front (93% “Save and Read Later”) and back
(84%). The design, which created visual hierarchy using a mixture of different font weights, logos, and
type-faces, featured a clear call to action on both sides of the reminder card.

Open-ended recall of the items in the Mail Sort Exercise
After the Mail Sort Exercises, participants were asked several open-ended questions to assess their
recall from the brief view through the eight mail pieces. The results were coded to identify trends
between the various mail packages; the full set of codes is included in Appendix B: In-depth Findings.
Below, we present the most pertinent findings from this analysis.
What kind of mail items do you remember seeing in the mail sort exercise?
(Coded open-ended responses, selected responses)
75%

Mentioned Census specifically
Mentioned government/official (no mention of Census)
69%
68%
65%

50%

40%

33%

25%
7%

6%

3%

0%
Official

Community

Mail Package

Patriotic

Control

Participants were randomly divided in half to measure aided awareness. One half was asked
specifically if they recalled seeing anything from the federal government. The other half was asked if
they recalled seeing anything from the Census Bureau. In both groups, participants who indicated
“yes” were asked to describe what they recalled seeing in an open-ended format, which was
subsequently coded.
Do you recall seeing anything from the federal government? (Asked of half)
Official

91%

Community

% Yes

87%

Patriotic

94%

Control

89%
0%

25%

50%

75%

100%

About nine in ten participants recalled seeing something from the federal government for each mail
package. However, when asked to describe what they recall seeing from the federal government, the
Official, Community, and Patriotic mail designs were much more likely than the Control to mention the
Census Bureau, rather than the Commerce Department.

25

Online Visual Testing Report
OF THOSE WHO SAID YES: What do you recall seeing from the federal government?
Mentioned Census
Mentioned Commerce Dept.
Mail Package
Official
91%
1%
Community
96%
0%
Patriotic
91%
1%
Control
54%
29%

A similar pattern was seen among the other half of participants who were asked whether they recall
seeing something from the Census Bureau. More than nineteen in twenty of the Official, Community,
and Patriotic treatment groups said they saw something from the Census Bureau. On the other hand,
only four in five (81%) Control participants said they recalled seeing something from the Census Bureau.
Do you recall seeing anything from the Census Bureau? (Asked of half)
Official

95%

Community

96%

Patriotic

96%

Control

% Yes

81%
0%

25%

50%

75%

100%

Based on the open-ended recall questions, participants were less likely to recognize that the pre-notice
envelope, Internet invitation envelope, and reminder postcard as coming from the U.S. Census Bureau.
This may be because the Control package placed the Census Bureau logo in the bottom corners on
envelopes, and put the Census Bureau name on the envelope under two lesser known agencies (U.S.
Commerce Department and the Economics and Statistics Administration).
The higher identification in the Control design with the federal government and Commerce Department
is potentially concerning. In our 2014 Refinement Messaging survey, we observed that the Census
Bureau has significantly more favorable name recognition among the public we interviewed than the
Commerce Department. Furthermore, the positive views towards the Census Bureau were less
correlated with negative views about the federal government than with the Commerce Department
(Hagedorn & Green, May 2014).
Official Internet Invitation Envelope (A1/C1)

Control Internet Invitation Envelope (A1/C1)

26

Online Visual Testing Report
There is an additional finding stemming from the follow-up coding that is worth noting. When
respondents were asked what they recalled seeing from the Census Bureau, one in eight (13%) who saw
the Official design mentioned the items should be opened immediately in the open-ended responses.
Some example responses included descriptions like, “handle immediately,” “open right away,” and
“something saying immediate response is required.” For the participants from the other mail packages
(including Control), almost no participants mentioned the need to respond immediately (2% or less,
p<.05).

Pre-notice Letter
After the Mail Sort Exercise, participants reviewed a pre-notice letter (Pre2) that was specific to the mail
package design. Participants first went through an image click exercise, and then went through message
highlighting. Afterwards, we asked participants some open-ended questions to measure their
understanding of the letter and probe them as to any areas they found confusing.

Image Click Analyzer #1: Pre-notice Letter
In the Image Click Analyzer, participants clicked three times on the image of the letter in the areas of the
letter that first caught their attention. The Official and Patriotic pre-notice letters have definitive visual
hierarchies, indicated by the largest proportion of clicks moving from one part of the letter to another.
In the Official pre-notice letter, two in three (67%) of participants clicked on the US Census logo for their
first click, and nearly half (45%) clicked inside the “Your response is required by law” box for their
second click. The third click was more evenly distributed, with the logo (23%), required box (21%),
respondent’s addresses towards the top (17%), and the link to the ACS website (16%) all having upticks.
The Patriotic pre-notice letter has two visual elements that attract the largest share of the first two
clicks. On the first click, the logo received a majority (54%) of clicks, with the “Your response is required
by law” in a bolded, all caps, box receiving another 33%. For the second click, it was essentially reversed,
with 49% of clicks in the required box and 24% in the logo. On the final click, the participants’ attention
was more disbursed, with elements like the instructions to look out for the survey (15%) and the ACS
website link (14%) receiving a portion of the attention.

27

Pre-notice

1st Click

2nd Click

Official

Community

Patriotic

Control

28

3rd Click

Online Visual Testing

The other two pre-notice letters did not have dominant visual hierarchies, though the logos and return
addresses at the top of those letters received considerable attention from participants.
The Community pre-notice letter’s logo was the dominant portion of the picture; seven in ten (71%)
participants clicked that logo for their initial click. As seen in the chart above, many of the clicks traveled
from the US Census Bureau portion to the orange “American Community Survey” portion of the logo,
with 29% of second clicks also inside the logo. The next highest area was the “Your response is required
by law” text, with 21% of clicks. Farther behind was the firefighter image and the instructions to “Please
look out for this information in the mail and complete this important survey as soon as possible,” which
suggests that at least two of the elements this design is meant to highlight are being overlooked.
The Control pre-notice letter began with 74% of participants indicating that their eye was initially caught
by the return address in the top right and an additional 8% on the Commerce Department logo next to
the Census Bureau’s address. On the second click, no area was a clear winner: 25% clicked on the
address the letter is mailed to (the highest proportion among the return addresses tested), and nearly
four in ten (38%) clicked on the Commerce Department logo or Census Bureau address.

Message Highlighting #1: Pre-notice Letter
After the Image Click Analyzer, participants used a Message Highlighting tool that focused on the words
and phrases in the letter. Participants selected up to forty words that were compelling in the letter,
allowing the research team to identify trends in the content that was highlighted. To help illustrate
which words and phrases received a significant portion of the selections, we created re-scaled message
maps that use larger and more red text to indicate words that are frequently highlighted. The full wordby-word data for each letter can be found in Appendix B: In-depth Findings.
Please note there is a small discrepancy between Control prenotice letter in the image click analyzer
(the ACS letter that did not feature a director’s signature) and the letter that participants viewed to
highlight in the Message Highlighting, which included a signature from Census Director John Thompson
and a slightly different salutation. We do not believe the difference has an impact on our findings.
As illustrated below, the sentence “Your response is required by law” was by far the most frequently
highlighted words in the three alternative mail package designs. At least three quarters (75%) of
participants in the Official, Community, and Patriotic pre-notice letters highlighted every word in that
sentence. In the Community and Patriotic pre-notice letter, the information about watching for the mail
(which was presented on a separate line and in bolded text) also received significant attention.

29

Online Visual Testing Report
Message Highlighting #1: Pre-notice letter
Official Pre-notice Letter

Patriotic Pre-notice Letter

Community Pre-notice Letter

Control Pre-notice Letter

*Note: color scales are reset for each message, and thus are not necessarily directly comparable across pre-notice letters. See Appendix B for full data.

The Control pre-notice letter does not have a required by law statement. It informs households they will
receive instructions for a “very important national survey” and that the “U.S. Census Bureau” is
conducting the survey. It is worth noting that two of the first three sentences use a passive voice
construction. The second paragraph describes how the survey is used to provide government services—
several sentences that received a moderate degree of attention from participants in the Message
Highlighting including education, housing, jobs, schools, highways, and hospitals. Participants seem to
notice the extended description. In a subsequent open-ended question (where participants are asked to
explain the purpose of the item they just saw to a friend), nearly half (47%) mention that the survey
helps determine the needs of the community or how funds or services are distributed. This is
significantly higher than responses from those who saw other designs (Official: 20%, Community: 21%,
Patriotic: 21%; p<.01).
Participants were also asked if there was anything unclear or confusing in the letter they read. Those
open-ended results were coded and grouped together; no topic had more than 4% of the responses.
30

Online Visual Testing Report

Internet Invitation Letter
The next mail item was the Internet invitation envelope (A1/C1) that was specific for each mail package
design. This style of large-sized envelope would be used to send the mail-back questionnaire.
Participants first went through an Image Click Exercise with either the front or back of the envelope, and
then answered questions about how well seven adjectives applied to the envelope they had just
reviewed.

Image Click Analyzer #2: Internet Invitation Envelope
On the next page, we have illustrated heat maps for the various Internet invitation envelopes. Largescale images are available in Appendix B: In-depth Findings.
In six of the seven Internet invitation letters, the Census Bureau logo received a majority of the 1st clicks.
The Census Bureau logo received the majority of clicks in all three locations where it was in the top left
(above the return address), both envelopes where it was centered on the top flap, and on the back
where it was in the top right corner. The exception was the Control envelope, where the logo was
smaller and placed in the lower left corner. For the Control envelope, the number of clicks increased on
the 2nd and 3rd round, suggesting that participants may have initially overlooked the Census Bureau logo
(similar to the findings in the Mail Sort Exercise).
The front of the Official Internet invitation envelope had a strong visual hierarchy. Nearly three in five
(57%) respondents clicked on the Census Bureau logo above the mailing address. On the second click,
half (49%) identified the “Your response is required by law” callout box. Finally, the largest portion (38%)
of respondents selected on the reverse text “OPEN IMMEDIATELY” box on the right side of the letter
with the third click.
On the front of the Community Internet invitation, the Census Bureau / ACS logo received considerable
attention, with 70% of the 1st clicks. The “Your response is required by law” notice received a majority
(54%) of the 2nd clicks. The Virginia icon received considerably less attention (at most, only 9% of the 3rd
clicks). Part of the underwhelming response may be due to the fact that the state image was not tailored
to respondents’ particular states. However, the lack of attention implies that the state image with
budget numbers may add more visual clutter rather than provide a compelling reason for completion of
the survey.
Respondents identified both the Census Bureau logo (54% of 1st clicks) and the “Your response is
required by law” (36% of 1st clicks) on the Patriotic Internet invitation front. In general, they switched
order for the 2nd click, with 42% identifying the required by law callout box and 33% clicking on the
Census Bureau logo. The final click was dispersed across the envelope.

31

st

1 Click

nd

2

Click

Community Front

Front
Front

Control Front*

Patriotic Back

Patriotic Front

Community Back

Official Back

Official Front

A1/C1

*note: the blank back of the control letter was not included in the image click exercise.

32

rd

3 Click

Online Visual Testing
On the Control Internet invitation, the “Your response is required by law” callout box received nearly
three in five (57%) of initial clicks. The return address (which did not have a Census Bureau logo)
received the largest portion of the second click (39%), though that proportion is substantially below the
attention paid to the top left corners of the other envelope fronts. That difference may have contributed
to differences in perceptions about the envelopes, which were explored in greater detail in the
subsequent attributes questions.

Internet invitation attributes and perceptions
After having an opportunity to view both sides of the envelope, participants were asked to indicate how
much they believed a series of seven adjectives applied to the envelope they just viewed.
Please indicate how much you believe the following words apply to the
envelope you just viewed.
1)
2)
3)
4)
5)

Applies completely
Applies somewhat
Applies a little
Does not apply at all
Don’t know

We observe some statistically significant differences between the mail packages on some terms. The
following table displays the percentage of respondents that indicated the adjective applied completely
to the envelope they had just reviewed. For all packages, more than three in five participants indicated
that “official” applied completely. Other terms, like “trustworthy” and “informative” had one in three or
fewer participants indicated they applied completely. For full results, see Appendix B: In-depth Findings.
% selected “Applies completely”

Descriptive Terms

(+/- compared to Control package)

ALL

Official

64

Important

56

Attention-grabbing

51

Easy-to-understand

46

Urgent

40

Trustworthy

31

Informative

28

Official

Community

Patriotic

Control

69

60

64

63

(+6)

(-3)

(+1)

(n/a)

65

53

58

48

(+17*)

(+5)

(+10*)

(n/a)

60

43

56

44

(+16*)

(-1)

(+12*)

(n/a)

49

43

51

41

(+8)

(+2)

(+10*)

(n/a)

58

30

39

33

(+25*)

(-3)

(+6)

(n/a)

30

31

34

28

(+2)

(+3)

(+6)

(n/a)

31

28

31

22

(+9*)

(+6)

(+9*)

(n/a)

*indicates statistically significant difference with family-wise correction for multiple comparisons (p<.05)

Compared to the control, the Official package appears especially successful at conveying the urgency of
the mail item with terms like “urgent” (+25 percentage points), “important” (+17), and “attentiongrabbing” (+16) all having differences in what completely applies to the mail package. From a design
standpoint, we attribute this difference to the “Open Immediately” callout box on the front of the
envelope, the governmental-looking Commerce Department logo, and the overall “Spartan” aesthetic.

33

Online Visual Testing Report
The following graph illustrates four adjectives with double-digit differences between the various
alternative designs and the Control package.
Please indicate how much you believe the following words apply to the envelope you just viewed.
70%

65%
60%

58%

60%
53%

% "Applies completely"

58%

56%

51%

49%

48%

50%

43%

44%

43%
39%

40%
30%

41%

Official

33%

Community

30%

Patriotic
Control

20%
10%
0%
Important

Attention-grabbing

Urgent

Easy-to-understand

Image Click Analyzer #2: Internet Invitation Letter
After examining the Internet invitation envelopes, respondents saw a text display informing them that
the next item was a look at a letter that would have been included in the envelope they just saw.
Participants then used an image click tool, followed by message highlighting, and some open-ended
questions. The following page depicts click heat maps for the four Internet invitation letters. Full-size
heat maps for each mail item can be found in Appendix B: In-depth Findings.
Similar to the pre-notice letters, the logos received more than half of the first clicks for each of the
designs (Official: 52%, Community: 73%. Patriotic: 62%, Control: 80% combining the Commerce
Department logo and return address).
In the Internet invitation letters, we tested three different treatments for the 1-800 telephone number
at the end of the letter. In each letter, the telephone number is included in a separate paragraph just
before the signature/closing. We find that having the telephone number in plain-text results in
somewhat lower number of clicks and selections in the message highlighting activity, though the
differences are not statistically significant.
Package

Text of sentence containing telephone number

Official
Community
Patriotic
Control

If you need help completing the survey or have questions, please call 1-800-354-7271.
If you need help completing the survey or have questions, please call 1-800-354-7271.
If you need help completing the survey or have questions, please call 1-800-354-7271.
If you need help completing the survey, please call our toll-free number (1-800-354-7271).

34

Message Highlighting
Image Click Analyzer
1st Click 2nd Click 3rd Click % selected phone #

0%
0%
0%
0%

0%
1%
0%
1%

1%
2%
3%
5%

6%
0%
4%
11%

Letter (A3)

Official

Community

Patriotic

Control

1st Click

2nd Click

3rd Click

Online Visual Testing
In each Internet invitation letter, we presented the request to visit the ACS website in four different
ways. In terms of raising self-response rates, the goal of the Internet invitation letter is to encourage
respondents to go to the website. In the table below, we present the percentage of respondents that
clicked on the link section of the website for each of the four visual testing letters. When looking at
these data, it is important to remember that these differences did not happen in isolation in our survey,
and there are a variety of differences between the letters that can influence the relative rates that
particular portions may be clicked.
Package

Official

Visual treatment

Bolded, enlarged text in a blue accent
box

Bolded, enlarged light-blue text with
Community the link on a separate line with larger

darker blue text
Patriotic

Blue accent box with large margins, link
in bolded and enlarged font in blue text

Control

Lead-in sentence in normal text. For the
link, a black and bolded text that is
centered on the letter

Text of sentence with ACS website address
Using the enclosed Internet instruction card, please
complete the survey online as soon as possible at:
http://respond.census.gov/acs

1st Click 2nd Click 3rd Click

32%

39%

17%

Using the enclosed card, please complete the survey online as soon
as possible at:
https://respond.census.gov/acs

14%

35%

22%

Complete the survey online at: https://respond.census.gov/acs

21%

46%

12%

11%

29%

26%

The U.S. Census Bureau recently sent a letter to your household about
the American Community Survey. Using the enclosed instructions,
please complete the survey online as soon as possible at:
https://respond.census.gov/acs

Both of the letters that used accent boxes had higher percentages of respondents select them in the 2nd
click. This suggests that the boxes help identify the call to action for participants in the letter. The
control letter, which uses the most traditional paragraph structure, had a lower peak number of clicks
(29%) than the more distinct approaches, implying that the control letter was not as successful at
attracting readers’ eyes. When viewing only the text of the letter without the visual design spacing or
accent boxes in the message highlighting exercise, there was not a statistically significant difference
between the mail packages (the Official letter had 39% of respondents highlight the URL, compared with
40% in the Community invitation letter and 46% in the Control).
Finally, the Patriotic letter included a footer message about how the ACS is used to distribute $400
billion dollars in federal funds each year. The other letters did not contain footers or the $400
billion/year figure. In the image click testing, one in five (20%) participants made their third click on the
footer paragraph. This suggests that footers could be a useful opportunity to deliver a closing message
or call to action.

Message Highlighting #2: Internet Invitation Letter
After the click test, participants were asked to highlight the words that they found most compelling in
the Internet invitation letter. Similar to the pre-notice letter, the majority of participants selected the

36

ACS Online Visual Testing
“your response required by law” statements. The following table illustrates this, with larger and brighter
text indicating that a larger proportion of participants highlighted it. Complete word-by-word results can
be found in Appendix B: In-depth Findings.
Message Highlighting #1: Pre-notice letter
Official Internet Invitation Letter

Patriotic Internet Invitation Letter^

^Please note the text in the message highlighting activity
did not include the text in the URL accent box.

Community Internet Invitation Letter

Control Internet Invitation Letter

*Note: color scales are reset for each message, and so are not necessarily directly comparable across Internet Invitation letters. See Appendix B for full data.

After the message highlighting, participants were asked an open-ended follow-up question. Half of
participants were asked how they would describe the purpose of the letter to a friend. After coding the
responses, the findings were broadly similar, with no statistically significant differences.
The other half shared if there was anything unclear or confusing about the letter. Respondents
identified questions about why the survey was required by law, or what would happen if they did not
respond as the most common kind of issue, though less than one in twenty participants mentioned the
issue for each mail design (between 3% and 5% for the four designs).
37

ACS Online Visual Testing

Image Click Analyzer #4: Internet/Mail Instruction Card OR Reminder Postcard
For their final set of activities, participants saw one of three mail items: the Internet instruction card
(A2), the mail instruction card (C2), or either the front or back of the reminder postcard (B1). They used
the image click analyzer and answered open-ended coding questions about the mail item.
(A2, C2)

Internet Instruction Card (A2)

Mail Instruction Card (C2)

Official design

st

1 Click

nd

2 Click

rd

3 Click

Community design

st

1 Click

2

nd

Click

rd

3 Click

38

ACS Online Visual Testing
(…continued)

Internet Instruction Card (A2)

Mail Instruction Card (C2)

Patriotic design

st

1 Click

nd

2 Click

rd

3 Click

Control design

st

1 Click

2

nd

Click

rd

3 Click

Full image click results for each mail item can be found in Appendix B, including percentages for each
region of the mail items.
39

ACS Online Visual Testing
The Internet instruction card features the laser-printed mailing address and authentication code, which
are positioned in the center of the card so as to appear through the envelope window. As a result, the
structure of the Internet instruction cards is designed to bring material above, below, or to the sides of
the mailing label. In the image click testing, we see a scattered visual hierarchy for all four designs. The
largest share of participants indicated that their eyes began in the top left of the card on the Census
Bureau logo (Official: 44%, Community: 51%, Patriotic: 42%, Control: 64%). Subsequently, we see clicks
fan out in a variety of directions—including the URL, the mailing address, instructions, and instructions
to keep the card with no part of any design receiving over 36% on the second click. This suggests for
design purposes that effective Internet instruction cards should aim to create an “L” shaped visual
hierarchy that starts in the top-left corner, and either travels down the side before running across the
bottom, or travels across the top of the card before going down the side.
The mail instruction cards had a more straightforward visual design; the three alternative mail designs
featured two columns and the Control design featured a single column.
We did not specifically assess whether participants recognized there were two options to respond to the
ACS. In the open-end responses, half of the participants were asked how they would describe the
purpose of the item they just saw to a friend. Those responses were then coded into a single code that
best captured the meaning of the individual response. While the number of cases is small (mean of ~56
per design), we see that the three designs that included mandatory response messages all had among
the top three codes (Official: 13%, Patriotic: 10%, Community: 9%). As indicated by the click test heat
maps, the callout boxes about “Your response is required by law” drive people’s attention and focus.

Image Click Analyzer #4: Reminder Postcard (B2)
Looking at the findings from the front and backs of the reminder postcard designs, we have three
observations. As with previous mail pieces, logos and “Your response is required by law” messaging
attracts a significant amount of attention from adults who handle the mail for their households.
For the back of the Patriotic postcard, we see a large portion (35%) of respondents identifying the
Lincoln image or the “Your response is required by law” (39%) as their first click. Subsequently, most
participants identified their eyes as moving down the card to URL (20% of 2nd clicks and 32% of final
clicks) or telephone number (1% of 2nd clicks and 11% of final clicks). As a result, many eyes seemingly
skip over the paragraph that connects the postcard to the earlier mail pieces.
When asked to describe the purpose of the mail item in an open-ended question, adults who saw the
Control reminder card were most likely to discuss how the ACS helps their community or determines
how federal funds are determined (15%), compared to lower numbers for the Official (9%), Community
(9%) or Patriotic (4%) designs. In particular, this suggests that the similar language about “communities
across the country depend on information the American Community Survey provides to make
improvements to roads, schools, hospitals, and more” were overlooked by survey participants.

40

Control Back*

Control Front*

Patriotic Back

Patriotic Front

Community Front
Front

Community Back
Front

Official Back

Official Front

B1
1 Click

st

2

nd

Click
3 Click

rd

Online Visual Testing

Closing Attributes
After going through the exercises, participants were asked to indicate how much they believed a series
of seven adjectives applied to the envelope they just viewed.
Thinking about all the mail items you viewed, please indicate how much you believe
the following words apply to the envelope you just viewed.
1)
2)
3)
4)
5)

Applies completely
Applies somewhat
Applies a little
Does not apply at all
Don’t know

We observed statistically significant differences between the mail packages on some terms, but not for
others. All the mail packages seemed roughly equally effective at conveying an “official” tone for the
mail. Similar to the initial envelopes, relatively smaller numbers of respondents said adjectives like
“trustworthy” and “informative” applied completely to the mail packages.
The following table displays the percentage of respondents that indicated the adjective applied
completely to the envelope they had just reviewed. For full results, see Appendix B: In-depth Findings.
% selected “Applies completely”

Descriptive Terms

(+/- compared to Control package)
ALL

Official

60

Important

53

Easy-to-understand

50

Attention-grabbing

47

Urgent

41

Trustworthy

36

Informative

32

Official

Community

Patriotic

64

56

62

Control

58

(+6)

(-2)

(+4)

(n/a)

59

50

56

44

(+15*)

(+6)

(+12*)

(n/a)

51

49

50

49

(+2)

(+0)

(+1)

(n/a)

55

43

50

39

(+16*)

(+4)

(+11*)

(n/a)

52

36

42

34

(+18*)

(+2)

(+8*)

(n/a)

39

36

36

35

(+4)

(+1)

(+1)

(n/a)

34

33

35

28

(+6)
(+5)
(+7*)
(n/a)
*indicates statistically significant difference with family-wise correction for multiple comparisons (p<.05)

After going through a series of mail items including envelopes and letters, the Official design appears
especially successful at conveying the urgency of the mail item with terms like “urgent” (+18 percentage
points), “important” (+15), and “attention-grabbing” (+16) all having differences in what completely
applies to the mail package. The high-contrast design elements like the “Open Immediately” callout box,
consistent use of the Census Bureau logo, and bolded text helps reinforce the urgency of the design.
The Patriotic design also had statistically significant differences with the Control in terms of timesensitivity. The Patriotic design was seen as being more “urgent” (+8 percentage points), “important”

42

ACS Online Visual Testing
(+12), and attention-grabbing (+11) than the Control. For all three measures, the Official design had a
larger effect than the Patriotic. The Patriotic design was seen as more “Informative” than the Control
design (+7), while the Official design just missed statistical significance.
By contrast, the Community design does not have any statistically significant differences from the
Control package. While some mail items, such as the back of the Internet invitation envelope had strong
visual hierarchies and tested strongly in the mail sort exercise, participants did not come away with the
same sense of vital significance as with the Official and Patriotic designs.
The following graph illustrates the seven descriptive terms that were tested after the visual exercises.
70%
64%

% "Applies completely"

60%

Thinking about all the mail items you viewed, please indicate how much you believe the
following words apply to the envelope you just viewed.
62%
.
59%

56%

58%

56%

55%
51%
49%50%49%

50%
50%

Official
Community
Patriotic
Control

44%

50%
43%

52%

42%
39%

40%

36%

34%

39%
36%36%35%

34%33%35%
28%

30%

20%

10%

0%
Official

Important

Easy-to-understand

Attention-grabbing

Urgent

Trustworthy

Informative

Closing Attributes
The survey included four attitude questions about the ACS. These questions were included in previous
Census Bureau studies, including the ACS Benchmark and Refinement messaging surveys, as well as
CBAMS (Bates et al., 2009).
ACS Favorability
ACS Likelihood to
Participate
Personal Benefit
and/or Harm
Community Benefit
and/or Harm

 Overall, how would you describe your general feelings about the American
Community Survey?
 How likely would you be to participate in the American Community Survey
(ACS) if contacted by the Census Bureau?
 Do you believe that answering the American Community Survey could
{ROTATED: personally benefit you in any way, personally harm you} or
neither benefit nor harm you?
 Thinking more generally, do you believe that answering the American
Community Survey could {ROTATED: benefit your community in any way,
harm your community}, or neither benefit nor harm your community?
43

ACS Online Visual Testing

We observed generally favorable views towards the American Community Survey, with 72% of mailhandling adults interviewed indicating they were very favorable or somewhat favorable. This number is
similar to the favorability the research team observed at the end of the telephone Refinement survey
(23% very favorable, 56% somewhat favorable, see Hagedorn & Green, 2014). There was little variation
between the four mail designs.
Overall, how would you describe your feelings about the American Community Survey?
Very favorable

27%

Somewhat favorable

45%

Somewhat unfavorable

10%

Very unfavorable

7%

Don't know

10%
0%

10%

(displaying all mail designs combined)
20%

30%

40%

50%

When asked how likely they would be to participate in the American Community Survey if contacted by
the Census Bureau, nearly 82% said they would be at least somewhat likely to participate.
We observed some difference in the intensity that participants believed they would participate; three in
five (61% each) mail-handling adults who saw the Community and Patriotic designs said they would be
very likely to participate. That is greater than those who said the same after seeing the Official design
(51%) or the Control (49%) (p<.05). Combining the “very likely” and “somewhat likely” categories into
one, we see almost no difference between the
How likely would you be to participate in the American
Community Survey if contacted by the Census Bureau? (by mail
mail designs, with all four designs between 79%
design)
and 85% of mail-handling adults interviewed
100%
4%
5%
5%
6%
selecting one of those two options. In light of the
6%
8%
8%
7%
4%
6%
5%
6%
strong sense of urgency conveyed by the Official
75%
Don't know
design as noted in the attribute questions, it is
24%
20%
29%
30%
somewhat curious that the perceived
Very unlikely
participation rate would not also be higher. This
Somewhat unlikely
50%
may be further evidence that stated intentions
Somewhat likely
to participate in a survey are not a good
Very likely
61%
61%
indicator of future behavior, similar to other
51%
25%
49%
studies that look at similar challenges with selfreported voting behavior (Rogers & Aida, 2012).
0%

Official

Community Patriotic

Control

We asked two questions to measure the
perceived impact respondents thought participating in the American Community Survey would have on
them personally and the impact on their communities. Similar to previous studies, we found that more
mail-handling adults we interviewed believe that answering the ACS could help their community (46%
benefit alone, 7% both benefit and harm), compared to the personal benefit of participating (26%
benefit alone, 5% both benefit and harm). Furthermore, the mail-handling adults we interviewed did not
44

ACS Online Visual Testing
think completing the ACS could harm their community (2% harm alone, 7% both benefit and harm) or
themselves (3% harm alone, 5% both benefit and harm).
Do you believe that answering the American Community
Survey could [benefit your community | harm your
community] or neither benefit nor harm your community?

Do you believe that answering the American Community
Survey could [personally benefit you | personally harm
you] or neither benefit nor harm you?

50%
50%

50%

40%

40%

47%

46%

Patriotic

Control

41%

30%

28%

28%

30%

23%

23%

20%

20%

10%

10%

0%

0%

Official

Community

Patriotic

Official

Control

Community

% selected "benefit my community"

% selected "personally benefit you"

We also observed a small, but not statistically significant, difference between the mail packages. The
Patriotic and Community designs had five percentage points more respondents say the survey could
benefit them personally than the Official or Control designs (28% vs. 23% respectively). We found a
similar trend with the question on whether the community benefits—while nearly half of the
Community (50%), Patriotic (47%), Control (46%) said answering the ACS could benefit their community,
a somewhat smaller percentage said the same about the Official package (41%). This suggests there may
be an opportunity to explore ways to include more language about the uses and users of ACS data in the
pre-notice and Internet instruction letter to present more information about how the ACS benefits
communities.

45

ACS Online Visual Testing

RECOMMENDATIONS
Based on the findings from the Online Visual Testing study, we present five recommendations to
improve the alternative mail package designs in anticipation of a real-world field test. We also offer
suggestions to improve Census Bureau communications about the ACS and explore further
opportunities as part of the alternative mail package development process.

Visual design elements and deadline messaging can have a significant impact on how
people read and remember mail items. The Online Visual Testing study found opportunities to
make the ACS mail package more eye-catching, important, and authoritative to households. After
viewing the Internet invitation envelopes, respondents who saw the Official design were 2.8 times more
likely to say the envelope was “urgent” than those who saw the Control (58% “applies completely” vs.
33%, p<.01). They were also more likely to agree it was “important” and “attention-grabbing.” As a
result, the front of the Official Internet invitation envelope was the top performer, with nineteen in
twenty (95%) mail-handling adults indicating they would “Save and Read Later” the envelope.

The U.S. Census Bureau logo should be prominently featured on mail items, where
appropriate. The Census Bureau logo is a powerful recognition tool. In the image click tool, the Census
Bureau logo received the majority of initial clicks in virtually all the click test heat maps, including the
envelopes where the image was on the right or center of the design.
In the Mail Sort Exercise, the pre-notice envelopes with the Census Bureau logo above the return address
had a five percentage point higher “Save and Read Later” rate than the Control envelope that did not
include the logo (p<.05). Furthermore, respondents who saw designs with prominent Census Bureau logos
were 27 percentage points more likely to specifically mention the “census” when asked what they
remembered seeing in the mail sort exercise. Those who saw the Control design, which did not have the
Census Bureau logo, were much more likely to mention the mail as coming from the Commerce
Department (29%) than the those who saw the alternative designs (1% or less) in open-ended responses.
As noted in previous studies, the Census Bureau has significantly higher favorability than the Commerce
Department or the federal government as a whole.

Key words and phrases in letters should be emphasized using callout boxes, line
spacing, and bolded text. In the Image Click Analyzer, we tested several different treatments for
Web addresses, telephone numbers, and reasons to respond. Elements that were emphasized using
graphic design techniques received more attention. For example, the Patriotic Internet invitation used a
blue accent box to call out the Web address to complete the survey: this element was clicked earlier and
more frequently in the image click exercise, compared to the same content when featured less
prominently in other designs, such as the Control.

46

ACS Online Visual Testing

“Your response is required by law” is the single most effective message to attract
readers’ attention. For envelopes, letters, instruction cards, and reminder notices, the mandatory
messaging clearly caught participants’ attention. For example, nine in ten participants highlighted the
words “required by law” in the Official pre-notice letter, which was more than three times greater than
the next most identified words. While some participants indicated in open-ended responses that they
didn’t understand why their response was required by law, these test results strongly support continued
use and further experimentation to continually improve mandatory message language on envelopes and
letters.

It is possible to overdo the commercial “marketing” look and feel. In the Mail Sort Exercise,
nearly one in three (31%) of respondents who initially saw the back of the Patriotic reminder postcard
(which featured a prominent Lincoln Memorial image) indicated that they would sort the postcard into
the “Trash.” Similarly, the respondents discarded the vast majority of the advertising pieces in the “clutter
reel” (department store ad: 80% “Trash,” dental insurance ad: 87% “Trash,” car ad: 95% “Trash.”) The
Census Bureau should be mindful of this threshold when designing further field testing. On the other
hand, designs that are formal yet evocative may be a catalyst for drawing respondents into the ACS
material. This is demonstrated by the Patriotic design’s use of color, a flag motif, and American
landmarks.

47

ACS Online Visual Testing

APPENDIX A: WORKS CITED
Bates, N. et al. (2009). “Messaging to America: Census Barriers, Attitudes, and Motivators Survey Research
(CBAMS).” American Association of Public Opinion Researchers (AAPOR) Conference Paper.
Hollywood, Florida.
Callegaro M. & C. Disorga. (2008). “Computing Response Metrics for Online Panels”. Public Opinion
Quarterly, v. 72, n. 5.
Chestnut J. (2010). Testing an Additional Mailing Piece in the American Community Survey. 2009 American
Community Survey Additional Mailing Test.
Groves R.. (1 March 2012). The Pros and Cons of Making the Census Bureau’s American Community Survey
Voluntary. Testimony before House Committee on Oversight and Government Reform.
Hagedorn, S and R. Green. (May 2014). “ACS Messaging Research: Refinement Survey.” American
Community Survey Research and Evaluation Report Memorandum Series.
Leslie, T. (13 Nov. 1996). “U.S. Census Test – Mail Response Analysis.” 1996 National Content Survey DSSD
Memorandum No. 2.
Matthews, B. et al. (25 June 2012). “2011 American Community Survey Internet Tests: Results from Second
Test in November 2011.” American Community Survey Research and Evaluation Program.
Nichols, E. (2012). “The April 2011 American Community Survey Internet Test: Attitudes and Behavior Study
Follow Up.” American Community Survey Research and Evaluation Program, #2012-03.
Olson, T. (2013). 2012 ACS Self Response Data. Respondent Advocate for Household Surveys.
Rogers, T. & Aida, M. (2012). “Why Bother Asking? The Limited Value of Self-Reported Vote Intention.”
Faculty Research Working Paper Series. Harvard Kennedy School.
Schwede, L. (2008). “Carrot or Stick Approach to Reminder Cards: What do Cognitive Respondents Think?”
Conference Paper, 2008 AAPOR: Section of Survey Research Methods.
Schwede L. & A. Sorokin (2009). “To Link or Not to Link? Exploring Approaches to Maintaining American
Community Survey Response Rates During Census 2010”. American Association of Public Opinion
Researchers (AAPOR)Conference Paper. Hollywood, Florida.
United States Census Bureau (April 2009). “American Community Survey: Design and Methodology.” US
Department of Commerce.
United States Census Bureau (2013). Current Population Survey, 2013 Annual Social and Economic
Supplement, Tables H1, H2, and H3. US Department of Commerce.

48

APPENDIX B: IN-DEPTH FINDINGS
Description of Activities
Exercise

# of sides

Mail Pieces Tested

Two ACS Mail pieces prenotice envelope (pre1) OR
Envelope (A1_C1) OR reminder postcard (D1) AND six
pieces of ‘clutter’ representing other mail items
Image click analyzer (#1) Prenotice letter (Pre2)
Message highlighting (#1) Prenotice letter (Pre2)
Mail stack exercise
(#1)

n-size
(per design/side)

2
(front/back)

1
1
2

333
500
500

Image click analyzer (#2)

Envelope (A1_C1)

Image click analyzer (#3)

Internet invitation letter (A3)

1

500

Message highlighting (#2) Internet invitation letter (A3)

1

500

Image click analyzer (#4)

(either front or back)

Instruction card (A2) OR Instruction card: choice (C2) OR
reminder postcard

2
(English-language
side; front/back)

Definition of Crosstabs
Crosstab
All
Official
Community
Package
Patriotic
Control

Definition of crosstab
All participants
Saw “Official” design for all mail exercises
Saw “Community” design for all mail exercises
Saw “Patriotic” design for all mail exercises
Saw “Control” design for all mail exercises

n-size
2,009
509
500
501
499

Four Mail Package Designs (A1_C1)
“Official”
“Community”

“Patriotic”

“Control”

49

250

% total
100
25
25
25
25

125

ACS Online Visual Testing

Demographic Quotas
We used quotas to balance age, gender, and racial/ethnic characteristics. The quotas used a crossdesign (age x gender x racial/ethnic group). The following table provides a detailed breakdown of the
demographic quotas for the survey:
Age
18-29
18-29
18-29
18-29
18-29
18-29
18-29
18-29
18-29
18-29
30-44
30-44
30-44
30-44
30-44
30-44
30-44
30-44
30-44
30-44
45-64
45-64
45-64
45-64
45-64
45-64
45-64
45-64
45-64
45-64
65+
65+
65+
65+
65+
65+
65+
65+
65+
65+

Gender
Male
Female
Male
Female
Male
Female
Male
Female
Male
Female
Male
Female
Male
Female
Male
Female
Male
Female
Male
Female
Male
Female
Male
Female
Male
Female
Male
Female
Male
Female
Male
Female
Male
Female
Male
Female
Male
Female
Male
Female

Race
White, non-His.
White, non-His.
Hispanic, any
Hispanic, any
Black, non-His.
Black, non-His.
Asian, non-His.
Asian, non-His.
Other/multiracial, non-His.
Other/multiracial, non-His.
White, non-His.
White, non-His.
Hispanic, any
Hispanic, any
Black, non-His.
Black, non-His.
Asian, non-His.
Asian, non-His.
Other/multiracial, non-His.
Other/multiracial, non-His.
White, non-His.
White, non-His.
Hispanic, any
Hispanic, any
Black, non-His.
Black, non-His.
Asian, non-His.
Asian, non-His.
Other/multiracial, non-His.
Other/multiracial, non-His.
White, non-His.
White, non-His.
Hispanic, any
Hispanic, any
Black, non-His.
Black, non-His.
Asian, non-His.
Asian, non-His.
Other/multiracial, non-His.
Other/multiracial, non-His.
Total*

“Official”
21
24
3
4
4
4
2
2
2
2
45
49
7
8
8
8
3
3
2
2
65
73
10
11
11
12
4
4
2
2
32
50
5
8
5
8
2
3
2
2
514

“Community”
21
24
3
4
4
4
2
2
2
2
45
49
7
8
8
8
3
3
2
2
65
73
10
11
11
12
4
4
2
2
32
50
5
8
5
8
2
3
2
2
514

*note totals add up to more than n=500 because of rounding and minimum cell size of two cases.

50

“Patriotic”
21
24
3
4
4
4
2
2
2
2
45
49
7
8
8
8
3
3
2
2
65
73
10
11
11
12
4
4
2
2
32
50
5
8
5
8
2
3
2
2
514

“Control”
21
24
3
4
4
4
2
2
2
2
45
49
7
8
8
8
3
3
2
2
65
73
10
11
11
12
4
4
2
2
32
50
5
8
5
8
2
3
2
2
514

Total
84
96
12
16
16
16
8
8
8
8
180
196
28
32
32
32
12
12
8
8
260
292
40
44
44
48
16
16
8
8
128
200
20
32
20
32
8
12
8
8
2056

ACS Online Visual Testing

Introduction and Screening Question
(ALL SAW THE FOLLOWING) PSB Interviewing, an independent research firm, is
conducting interviews to solicit feedback on potential methods to respond to research
surveys.
Participation is voluntary, but your responses are important. Answers will be kept
anonymous and the survey will not ask for information that could personally identify you.
At the end of the interview, you will be provided with an email address where you can
send any comments or questions about this survey. This survey has been approved by
U.S. Office of Management and Budget approval number 0607-0760, on behalf of the
U.S. Census Bureau, expiring on September 30th, 2014.
If you agree to participate, we will begin by asking you a few questions to see if you
qualify to participate in the study. If you are eligible, the survey will take approximately
20 minutes to complete.
1.

Do you generally handle the mail for your household?
(%)
Count (n-size)
Yes
No
Don't know

All
2009
100
*
*

Package
Official

Community

Patriotic

Control

509
100
*
*

500
100
*
*

501
100
*
*

499
100
*
*

*By definition, all participants had to handle the mail for their household.
Mail Stack Exercise
(ALL SAW THE FOLLOWING) Thank you. You qualify to participate in the study.
On the next screen, you will be asked to sort several mail items that you might see in
your household’s mail. After looking at each item, please click-and-drag the item to
indicate if you would "save and read later" or if you would put it in the "trash." You may
flip over the mail item by clicking at the top of the screen.
When you are ready, please click on the arrow at the bottom of the screen to load the
mail sort exercise.
2.

SELECT TWO ITEMS FOR MAIL SORT EXERCISE (CODED AT-RANDOM)
(%)
Count (n-size)
Pre-notice envelope
Envelope
Reminder postcard

All
2009
66
66
68

51

Package
Official

Community

Patriotic

Control

509
65
66
69

500
66
66
68

501
65
66
68

499
67
67
66

ACS Online Visual Testing

Image

% overall select
“Save and Read Later”

% Read
by Side

Flip Over

Median
View Time*

Pre1_Official

Among those who saw the front first:

93%

85% selected
“Save and Read Later”
(15% select “Trash”)

13%

selected
flipped over mail item
“Save and Read Later”
to view back side

F: 5.5 seconds
(s.d.: 5.4)

B: 1.0 second
(s.d.: 2.5)

Among those who saw the back first:

78%

64%

selected
flipped over mail item
“Save and Read Later”
to view front side

F: 5.5 seconds
(s.d.: 5.7)

B: 3.5 seconds
(s.d.: 4.4)

Pre1_Community

Among those who saw the front first:

91%

86% selected
“Save and Read Later”

14%

selected
flipped over mail item
“Save and Read Later”
to view back side

F: 5.5 seconds
(s.d.: 5.7)

B: 1.0 second
(s.d.: 2.7)

Among those who saw the back first:
(14% select “Trash”)

80%

64%

selected
flipped over mail item
“Save and Read Later”
to view front side

F: 6.0 seconds
(s.d.: 4.8)

B: 3.5 seconds
(s.d.: 4.8)

Pre1_Patriotic

Among those who saw the front first:

93%

88% selected
“Save and Read Later”
(12% selected “Trash”)

12%

selected
flipped over mail item
“Save and Read Later”
to view back side

F: 4.5 seconds
(s.d.: 5.3)

B: 1.5 seconds
(s.d.: 3.3)

Among those who saw the back first:

84%

68%

selected
flipped over mail item
“Save and Read Later”
to view front side

F: 5.5 seconds
(s.d.: 5.2)

B: 4.0 seconds
(s.d.: 4.6)

Pre1_Control

Among those who saw the front first:

87%

80% selected
“Save and Read Later”

14%

selected
flipped over mail item
“Save and Read Later”
to view back side

F: 6.5 seconds
(s.d.: 5.8)

B: 1.5 seconds
(s.d.: 3.5)

Among those who saw the back first:
(20% selected “Trash”)

73%

67%

selected
flipped over mail item
“Save and Read Later”
to view front side

*Median time among those who saw a particular side (excluding those who did not view the side).

52

F: 6.5 seconds
(s.d.: 5.3)

B: 3.5 seconds
(s.d.:4.0)

ACS Online Visual Testing

A1/C1_ Official

Image

% overall select
“Save and Read Later”

% Read
Median
Flip Over
View Time*
by Side
Among those who saw the front first:

95%
F

91% selected
“Save and Read Later”

10%

selected
flipped over mail item
“Save and Read Later”
to view back side

F: 5.0 seconds
(s.d.: 5.6)

B: 4.5 seconds
(s.d.: 6.2)

Among those who saw the back first:
(9% selected “Trash”)

86%

18%

selected
flipped over mail item
“Save and Read Later”
to view front side

B

F: 4.0 seconds
(s.d.: 5.3)

B: 5.75 seconds
(s.d.: 4.4)

A1/C1_Community

Among those who saw the front first:

90%
F

93% selected
“Save and Read Later”
(7% selected “Trash”)

10%

selected
flipped over mail item
“Save and Read Later”
to view back side

F: 5.0 seconds
(s.d.: 5.4)

B: 3.5 seconds
(s.d.: 5.7)

Among those who saw the back first:

96%

26%

selected
flipped over mail item
“Save and Read Later”
to view front side

B

F: 4.0 seconds
(s.d.: 4.2)

B: 4.0 seconds
(s.d.: 4.2)

A1/C1_Patriotic

Among those who saw the front first:

94%
F

94% selected
“Save and Read Later”
(6% selected “Trash”)

8%

selected
flipped over mail item
“Save and Read Later”
to view back side

F: 4.5 seconds
(s.d.: 4.4)

B: 3.0 seconds
(s.d.: 5.0)

Among those who saw the back first:

94%

18%

selected
flipped over mail item
“Save and Read Later”
to view front side

B

F: 4.0 seconds
(s.d.: 4.6)

B: 4.5 seconds
(s.d.: 3.9)

A1/C1_Control

Among those who saw the front first:

92%
F

86% selected
“Save and Read Later”
(14% selected “Trash”)

18%

selected
flipped over mail item
“Save and Read Later”
to view back side

(s.d.: 5.3)

B: 1.5 seconds
(s.d.: 3.4)

Among those who saw the back first:

78%

67%

selected
flipped over mail item
“Save and Read Later”
to view front side

B
*Median time among those who saw a particular side (excluding those who did not view the side).

53

F: 7.0 seconds

F: 7.5 seconds
(s.d.: 5.0)

B: 4.0 seconds
(s.d.: 4.7)

ACS Online Visual Testing

Image

% overall select
“Save and Read Later”

% Read
by Side

Flip Over

Median
View Time*

Among those who saw the front first:

B1_ Official

93%

89% selected
“Save and Read Later”

F

12%

selected
flipped over mail item
“Save and Read Later”
to view back side

F: 4.5 seconds
(s.d.: 4.6)

B: 5.5 seconds
(s.d.: 6.8)

Among those who saw the back first:
(11% selected “Trash”)

84%

17%

selected
flipped over mail item
“Save and Read Later”
to view front side

B

F: 6.0 seconds
(s.d.: 3.9)

B: 7.0 seconds
(s.d.: 6.6)

B1_Community

Among those who saw the front first:

90%

84% selected
“Save and Read Later”

F

13%

selected
flipped over mail item
“Save and Read Later”
to view back side

F: 5.0 seconds
(s.d.: 4.2)

B: 5.0 seconds
(s.d.: 8.2)

Among those who saw the back first:

(16% selected “Trash”)

79%

11%

selected
flipped over mail item
“Save and Read Later”
to view front side

F: 3.0 seconds
(s.d.: 4.7)

B: 7.0 seconds
(s.d.: 5.8)

B
Among those who saw the front first:

B1_Patriotic

91%

80% selected
“Save and Read Later”

F

9%

selected
flipped over mail item
“Save and Read Later”
to view back side

F: 5.0 seconds
(s.d.: 4.3)

B: 7.0 seconds
(s.d.: 7.2)

Among those who saw the back first:

(20% selected “Trash”)

69%

14%

selected
flipped over mail item
“Save and Read Later”
to view front side

F:4.0 seconds
(s.d.: 5.2)

B: 7.0 seconds
(s.d.: 7.0)

B

B1_Control

Among those who saw the front first

91%

83% selected
F

“Save and Read Later”
(17% selected “Trash”)

19%

selected
flipped over mail item
“Save and Read Later”
to view back side

10%

selected
flipped over mail item
“Save and Read Later”
to view front side

*Median time among those who saw a particular side (excluding those who did not view the side).

54

(s.d.: 4.8)

B: 9.0 seconds
(s.d.: 7.8)

Among those who saw the back first:

77%
B

F: 6.0 seconds

F: 2.5 seconds
(s.d.: 1.4)

B: 9.0 seconds
(s.d.: 6.1)

ACS Online Visual Testing

Image

% overall select
“Save and Read Later”

% Read
by Side

Flip Over

Median
View Time*

X1: EDA BILL

Among those who saw the front first:

74%

70% select
“Save and Read Later”

F

(30% select “Trash”)

15%

selected
flipped over mail item
“Save and Read Later”
to view back side

F: 8.5 seconds
(s.d.: 6.5)

B: 1.5 seconds
(s.d.: 2.5)

Among those who saw the back first:

66%

61%

selected
flipped over mail item
“Save and Read Later”
to view front side

B

F: 9.0 seconds
(s.d.: 5.8)

B: 4.0 seconds
(s.d.: 4.2)

X2: Water Bill

Among those who saw the front first:

87%

86% select
“Save and Read Later”

F

(14% select “Trash”)

12%

selected
flipped over mail item
“Save and Read Later”
to view back side

F: 4.5 seconds
(s.d.: 5.0)

B: 3.0 seconds
(s.d.: 3.7)

Among those who saw the back first:

86%

47%

selected
flipped over mail item
“Save and Read Later”
to view front side

B

F: 4.0 seconds
(s.d.: 4.4)

B: 5.0 seconds
(s.d.: 4.6)

X3: Dental Ad

Among those who saw the front first:

9%
F

13% selected
“Save and Read Later”
(87% selected “Trash”)

9%

selected
flipped over mail item
“Save and Read Later”
to view back side

F: 3.5 seconds
(s.d.: 3.7)

B: 4.5 seconds
(s.d.: 5.2)

Among those who saw the back first:

17%

11%

selected
flipped over mail item
“Save and Read Later”
to view front side

B

F: 3.0 seconds
(s.d.: 4.0)

B: 5.0 seconds
(s.d.: 4.6)

X4: Retail Ad

Among those who saw the front first:

25%
F

20% selected
“Save and Read Later”

12%

selected
flipped over mail item
“Save and Read Later”
to view back side

F: 3.5 seconds
(s.d.: 3.8)

B: 4.5 seconds
(s.d.: 4.7)

Among those who saw the back first:
(80% selected “Trash”)

16%

11%

selected
flipped over mail item
“Save and Read Later”
to view front side

B
*Median time among those who saw a particular side (excluding those who did not view the side).

55

F: 3.5 seconds
(s.d.: 3.6)

B: 5.5 seconds
(s.d.: 5.0)

ACS Online Visual Testing
% Read
by Side

% overall select
“Save and Read Later”

Image

Flip Over

Median
View Time*

X5: Hand-written card

Among those who saw the front first:

90%

85% select
“Save and Read Later”

F

15%

selected
flipped over mail item
“Save and Read Later”
to view back side

F: 5.5 seconds
(s.d.: 5.2)

B: 1.5 seconds
(s.d.: 4.8)

Among those who saw the back first:

(15% select “Trash”)

80%

64%

selected
flipped over mail item
“Save and Read Later”
to view front side

B

F: 6.0 seconds
(s.d.: 4.9)

B: 4.0 seconds
(s.d.: 4.9)

Among those who saw the front first:

X6: Auto Ad

5%

5% select
“Save and Read Later”

F

12%

selected
flipped over mail item
“Save and Read Later”
to view back side

F: 3.5 seconds
(s.d.: 3.4)

B: 4.5 seconds
(s.d.: 5.1)

Among those who saw the back first:
(95% select “Trash”)

5%

F: 3.0 seconds

9%

selected
flipped over mail item
“Save and Read Later”
to view front side

(s.d.: 3.9)

B: 4.0 seconds
(s.d.: 4.6)

B
*Median time among those who saw a particular side (excluding those who did not view the side).

3.

What kind of mail items do you remember seeing in the mail sort exercise?
(OPEN-ENDED, CODED, MULTIPLE-CODES PERMITTED)

(%)

All

Count (n-size)

1960

61
46
44
39
36
35
33
32
12
9
9
6
2
7
1

Mentioned Census
Personal letter/card / Letter / Card
Abelcar / Flier for cars
Macy's Swimsuit ad / 20% off / Coupons
Statement / Bills (not specific)
Aetna Dental ad
Southwest Water Company bill
Advertising
Government/official mail (no mention of Census)
EDA bill / Financial statement / Lawyer's bill
Junk mail
Blank envelope(s) / Back of envelope(s)
Survey (unspecific)
Others
None / Don't Know / No Response

56

Package
Official

Community

Patriotic

Control

496
68
44
44
38
31
34
35
31
7
8
11
5
1
7
1

491
69
47
48
40
36
38
32
30
3
10
8
6
1
6
1

491
65
47
43
37
37
34
34
33
6
9
9
5
2
6
0

482
40
47
43
40
39
32
32
35
33
7
7
6
3
9
1

ACS Online Visual Testing
(PARTICIPANTS HEARD EITHER QUESTION 4 OR QUESTION 6)

4.

(HALF HEARD) Did you recall seeing anything from the federal government?
(%)

All

Count (n-size)

1027

90
5
4

Yes
No
Don't know

5.

Package
Official

Community

Patriotic

Control

258
91
3
6

242
87
5
7

251
94
3
2

276
89
8
3

(IF YES) What do you recall seeing from the federal government?
(OPEN-END, CODED, MULTIPLE-CODES PERMITTED)

(%)

All

Count (n-size)

Community

Patriotic

Control

83
11
9
9
8
7
4
4
4
3
2
2
1
1

234
91
8
6
9
1
8
3
4
5
1
1
2
0
0

206
96
11
6
13
0
9
2
2
4
3
2
1
0
0

235
91
13
12
8
1
7
3
3
6
3
1
2
0
1

239
54
13
11
8
29
4
5
6
3
4
3
4
1
4

0

0

0

0

1

3
5

2
3

1
4

2
4

6
8

914

Census
Item(s) / Mailer(s)
Request to fill out survey
Letter(s)
Commerce Department
Document(s) / Report(s)
Required by law / Mandatory
Postcard(s)
Form(s) to fill out
Reminder/follow up
Notice(s)
Envelope(s)
IRS
Federal Government / Government
American Community Survey (ACS) / Community
Survey
Don't Know / No Response
Others

57

Package
Official

ACS Online Visual Testing

6.

(HALF HEARD) Did you recall seeing anything from the Census Bureau?
(%)

All

Count (n-size)
Yes
No
Don't know

7.

982
92
4
2

Package
Official

Community

Patriotic

Control

251
95
2
1

258
96
2
2

250
96
3
1

223
81
11
5

(IF YES) What do you recall seeing from the Census Bureau?
(OPEN-END, CODED, MULTIPLE-CODES PERMITTED)

(%)

All

Count (n-size)

Request to fill out survey
Letter(s)
Required by law / Mandatory
Census / The name
Document(s) / Paper(s) / Material(s)
Envelope(s)
Postcard(s) / Card(s)
Reminder/follow-up
Open immediately
Important
Government/official-looking
Notice(s) / Notification
Addressed to resident
Should have recently received / Will be receiving
soon
Go to website / Complete online
Commerce Department
Junk mail
American Community Survey (ACS) / Community
Survey
Other
Don't Know / Haven't read contents yet

58

Package
Official

Community

Patriotic

Control

878

231

240

236

171

26
21
19
18
18
14
9
5
4
4
3
3
3

24
23
16
17
22
11
9
3
13
4
3
2
3

26
16
25
19
21
16
6
5
1
5
2
3
2

28
21
19
17
15
16
9
4
2
3
4
3
2

26
24
17
16
15
13
13
9
1
3
4
2
6

2

1

1

1

5

2
1
1

1
0
1

3
0
0

3
0
1

2
4
1

1

0

2

0

1

10
6

5
6

11
7

11
6

15
5

ACS Online Visual Testing

Image Click #1 (Pre-notice Letter)
(ALL SAW THE FOLLOWING) Now we will have you take a closer look at a letter that would be
inside one of the envelopes that you just saw. First, we will ask about your initial visual
impression of the letter. Then we will ask about the content of the letter.
8.

Pre-notice letter (Official)

(INSTRUCTIONS) Please click on the image below in areas that caught your attention when
you first looked at this letter. This could be pictures, words, logos, etc. Please click three times.

(%)
Logo
Address
Greeting
Paragraph 1
Required
Paragraph 2
Paragraph 3
Link
Signature
Footer

Click
1st Click

2nd Click

3rd Click

67%
3%
1%
1%
27%
0%
<1%
1%
<1%
<1%

33%
11%
1%
2%
45%
1%
1%
2%
1%
2%

23%
17%
4%
5%
21%
3%
2%
16%
3%
6%

Pre2_Official

1st Click

59

ACS Online Visual Testing

2nd Click

3rd Click

60

ACS Online Visual Testing

“Official” Pre-notice Message Highlighting #1
(INSTRUCTIONS) Now looking closer at the content of the letter, please identify the words or
phrases that you find most compelling by highlighting that portion of the text by clicking and
dragging your cursor over the words. You may highlight up to 40 words. After you are finished,
click on the arrow at the bottom of the screen to continue.

61

ACS Online Visual Testing

TO
9%
Apt
6%
Anytown,
4%

THE
9%
2,
6%
MD
4%

RESIDENT
12%
2147
6%
03612-1383
4%

OF:
9%
Montgomery
7%

Dr.
6%

A
18%

message
21%

from
20%

the
21%

U.S.
35%

Census
37%

Bureau:
37%

In
12%
to
10%
survey
7%
selected
7%

a
12%
complete
11%
and
5%
sample.
6%

few
14%
the
12%
chose
9%

days,
15%
American
19%
your
9%

you
13%
Community
20%
address,
11%

will
13%
Survey.
20%
not
10%

receive
16%
The
6%
you
10%

instructions
17%
U.S.
9%
personally,
9%

in
11%
Census
9%
as
4%

the
11%
Bureau
8%
part
4%

mail
12%
is
4%
of
4%

Your
83%
mail,
19%

response
88%
please
22%

is
83%
respond
25%

required
92%
as
25%

by
88%
soon
26%

law.
92%
as
26%

When
20%
possible.
26%

this
19%

important
20%

survey
20%

arrives
19%

in
19%

the
19%

Completing
22%
have
5%

the
21%
access
5%

survey
25%
to
4%

online
29%
the
4%

is
20%
Internet,
6%

fastest
23%
a
5%

and
17%
paper
7%

most
18%
survey
7%

efficient.
20%
will
5%

If
5%
be
5%

you
5%
mailed
6%

do
5%
to
4%

not
5%
you.
4%

Federal,
8%
Survey
8%
bringing
8%
promptly.
7%

tribal,
8%
to
5%
improvements
11%

state,
8%
understand
7%
to
8%

and
5%
and
6%
highways,
11%

local
7%
meet
8%
hospitals,
11%

leaders
8%
the
7%
and
7%

rely
8%
needs
9%
schools—so
11%

on
6%
of
7%
it
3%

To
5%
Web
9%

learn
5%
site:
13%

more
5%
https://census....
36%

about
5%

the
5%

Thank
1%

you
1%

in
1%

advance
1%

for
1%

John
1%
Director,
13%

H.
1%
U.S.
15%

Thompson
1%
Census
16%

Bureau
15%

Official Pre-notice Letter
Word-by-Word Messaging Highlighting
Frequency Chart

American Community
8%
8%

your
1%

prompt
2%

62

Survey,
8%

response.
2%

information
from
7%
5%
communities across
8%
6%
is
important
3%
4%

please
7%

visit
7%

the
5%
the
6%
that
4%

the
7%

about
how
9%
10%
conducting
this
5%
4%
a
randomly
5%
8%

American Community
8%
9%
nation— including
8%
8%
you
respond
4%
6%

Census
9%

Bureau’s
9%

ACS Online Visual Testing
9.

If you had to explain the purpose of the item you just saw to a friend, how would you
describe it to them? (CODED OPEN-ENDS)

Pre-notice letter (Official)
Count (n-size)

488

Letter from the Census Bureau (mentioned Census specifically)
Survey helps determine the needs of my community / How funds are allocated
Mandatory / Required by law
Survey / Reminder to fill out a survey (did not mention Census)
Randomly selected to do the survey
Explaining the importance of filling out the survey/why they are conducting the survey
Online survey / Important to complete online
More than one way to fill out the survey
Letter about previous or soon to come letter / Instructions coming soon
Not legitimate / A scam / Not part of the regular census
Junk mail / Spam / Garbage / Nonsense
Unnecessary / Useless
Government snooping / Using threats / Intimidation
All other negatives
Voluntary
Others
Don't Know / No Response

29%
20%
14%
6%
5%
4%
4%
4%
2%
1%
1%
1%
1%
1%
0%
2%
3%

63

ACS Online Visual Testing
10.

Thinking about the item you just read, were there any parts you found to be unclear or
confusing? (CODED OPEN-ENDS)

Pre-notice letter (Official)
Count (n-size)
It was fine / Nothing was unclear
Why is this required by law / What happens if I do not respond
What is the survey about / Need to see the real survey
Everything was unclear
Confusion about how to respond / response mode
Is it legitimate / Looks like junk mail
Letter was too vague / Was not sure what the letter was about
Who is this organization / Never heard of ACS
Addressed to 'resident' / Not addressed to me personally
Waste of money sending multiple mailings
Who benefits from my information / Does my information really help
How will the funds be distributed / Will the data be used
How was I 'Randomly selected'
When is it due / How long do we get to complete it
Cluttered / Too many fonts/sizes/colors / Writing too small to read easily
Others
Don't Know / Did Not Read It All / No Response

64

463
79%
4%
3%
2%
2%
2%
1%
1%
1%
1%
<1%
<1%
<1%
<1%
<1%
1%
2%

ACS Online Visual Testing

11.

Pre-notice letter (Community)

(INSTRUCTIONS) Please click on the image below in areas that caught your attention when
you first looked at this letter. This could be pictures, words, logos, etc. Please click three times.

(%)
Logo
Print Code
Address
Paragraph 1
Paragraph 2
Paragraph 3
Image (Firefighter)
Quote
Paragraph 4
Link
Signature
Footer

Click
st

1 Click

2nd Click

3rd Click

71%
<1%
3%
<1%
6%
2%
7%
1%
5%
1%
1%
<1%

29%
0%
13%
3%
15%
1%
12%
1%
21%
3%
1%
1%

19%
<1%
11%
3%
18%
1%
11%
2%
18%
12%
4%
2%

Pre2_Community

1st Click

65

ACS Online Visual Testing

2nd Click

3rd Click

66

ACS Online Visual Testing

“Community” Pre-notice Message Highlighting #1
(INSTRUCTIONS) Now looking closer at the content of the letter, please identify the words or
phrases that you find most compelling by highlighting that portion of the text by clicking and
dragging your cursor over the words. You may highlight up to 40 words. After you are finished,
click on the arrow at the bottom of the screen to continue.

67

ACS Online Visual Testing
U.S.
42%
Washington,
12%

Census
45%
DC
10%

TO
6%
Apt
6%
Anytown,
5%

THE
6%
2,
6%
MD
5%

In
14%
in
14%

a
15%
the
14%

few
16%
mail
15%

days,
16%
about
12%

Please
39%
survey
47%

look
40%
as
45%

out
40%
soon
48%

for
39%
as
46%

The
4%
service.
11%
and
5%
schools,
9%

American
6%
Federal,
6%
meet
7%
and
6%

Community
6%
tribal,
6%
the
6%
hospitals.
9%

Survey
6%
state,
5%
needs
8%

asks
4%
and
4%
of
6%

required
85%
receive
6%
asked
5%

by
82%
the
5%
to
5%

about
4%

the
3%

for
2%

Your
77%
address—not
16%
You
5%

Bureau
43%
20233
8%

Community Pre-notice Letter
Message Highlighting
Word-by-Word Frequency Chart

RESIDENT
OF:
7%
6%
2147
Montgomery
6%
6%
03612-1383
4%

response
is
80%
77%
you
personally—to
14%
13%
will
be
5%
5%

To
3%
Web
7%

learn
more
4%
4%
site: https://census...
9%
25%

Thank
3%

you
3%

in
2%

advance
2%

John
1%
Director,
6%

H.
<1%
U.S.
5%

Thompson
1%
Census
6%

Bureau
6%

“Our
2%
where
2%
—John
1%

county
2%
we
2%
Smith
<1%

looked
2%
can
2%
Fire
1%

to
2%
do
2%
Chief,
0%

Dr.
5%

the
15%
how
14%

U.S.
21%
to
14%

this information
39%
40%
possible.
47%

Census
22%
complete
16%

Bureau
22%
the
15%

in
38%

the
38%

about
topics
4%
7%
local
leaders
5%
6%
communities across
7%
5%

law.
84%
survey
6%
provide
6%

mail
39%

and
36%

sending
18%
Survey.
22%

you
16%

information
18%

complete
48%

this
44%

important
48%

and
8%
to
5%
to
6%

military
11%
understand
6%
roads,
9%

like
school enrollment, occupation,
6%
11%
11%
11%
rely
on
this
information
6%
5%
5%
7%
America, including bringing improvements
5%
4%
6%
9%

The
Census
Bureau
12%
12%
11%
as
part
of
5%
5%
5%
important information on
6%
6%
5%

American Community Survey,
5%
5%
5%

your
2%

will
be
16%
16%
American Community
22%
22%

has
11%
a
4%
behalf
5%

randomly
16%
nationally
6%
of
5%

selected
your
16%
14%
representative sample.
5%
5%
your
community.
5%
5%

please
4%

visit
5%

the
5%

Census
7%

Bureau’s
8%

to
2%
the
2%

locate
2%
most
2%

our
2%
people.”
2%

fire
2%

station
2%

response.
2%

Sincerely,
<1%

American Community
3%
3%
the
most
2%
2%
VA
1%

Survey
2%
good
2%

12

68

data
2%
for
2%

ACS Online Visual Testing
12.

If you had to explain the purpose of the item you just saw to a friend, how would you
describe it to them? (CODED OPEN-ENDS)

Pre-notice letter (Community)
Count (n-size)

477

Letter from the Census Bureau (mentioned Census specifically)
Mandatory / Required by law
Survey helps determine the needs of my community / How funds are allocated
Survey / Reminder to fill out a survey (did not mention Census)
Randomly selected to do the survey
Explaining the importance of filling out the survey/why they are conducting the survey
Letter about previous or soon to come letter / Instructions coming soon
Government snooping / Using threats / Intimidation
Junk mail / Spam / Garbage / Nonsense
All other negatives
Online survey / Important to complete online
Voluntary
Not legitimate / A scam / Not part of the regular census
Unnecessary / Useless
Others
Don't Know / No Response

31%
22%
21%
8%
5%
3%
2%
2%
1%
1%
<1%
<1%
<1%
<1%
1%
3%

69

ACS Online Visual Testing
13.

Thinking about the item you just read, were there any parts you found to be unclear or
confusing? (CODED OPEN-ENDS)

Pre-notice letter (Community)
Count (n-size)
It was fine / Nothing was unclear
Why is this required by law / What happens if I do not respond
What is the survey about / Need to see the real survey
Everything was unclear
How was I 'Randomly selected'
Is it legitimate / Looks like junk mail
Cluttered / Too many fonts/sizes/colors / Writing too small to read easily
Letter was too vague / Was not sure what the letter was about
Who benefits from my information / Does my information really help
How will the funds be distributed / Will the data be used
When is it due / How long do we get to complete it
Who is this organization / Never heard of ACS
Waste of money sending multiple mailings
Others
Don't Know / Did Not Read It All / No Response

70

450
82%
4%
3%
2%
1%
1%
1%
<1%
<1%
<1%
<1%
<1%
<1%
2%
3%

ACS Online Visual Testing

14.

Pre-notice letter (Patriotic)

(INSTRUCTIONS) Please click on the image below in areas that caught your attention when
you first looked at this letter. This could be pictures, words, logos, etc. Please click three times.
Click

(%)

st

Logo
Header (We The People)
Address
Greeting
Paragraph 1
Paragraph 2
Paragraph 3
Required
Paragraph 4
Paragraph 5
Signature
Print Code
Paragraph 6
Footer

1 Click

2nd Click

3rd Click

54%
3%
2%
1%
1%
3%
<1%
33%
1%
0%
1%
0%
1%
1%

24%
2%
6%
1%
2%
7%
1%
49%
1%
1%
1%
0%
3%
2%

11%
2%
14%
3%
3%
15%
1%
19%
2%
6%
3%
0%
14%
8%

Pre2_Patriotic

1st Click

71

ACS Online Visual Testing

2nd Click

3rd Click

72

ACS Online Visual Testing

“Patriotic” Pre-notice Message Highlighting #1
(INSTRUCTIONS) Now looking closer at the content of the letter, please identify the words or
phrases that you find most compelling by highlighting that portion of the text by clicking and
dragging your cursor over the words. You may highlight up to 40 words. After you are finished,
click on the arrow at the bottom of the screen to continue.

73

ACS Online Visual Testing
TO
6%
Apt
5%
Anytown,
4%

THE
6%
2,
5%
MD
4%

RESIDENT
7%
2147
5%
03612-1383
4%

OF:
6%
Montgomery
5%

From
21%

the
21%

U.S.
27%

Census
28%

Bureau:
27%

In
11%
in
11%

a
11%
the
11%

few
13%
mail
12%

days,
13%
about
9%

the
12%
how
10%

U.S.
15%
to
10%

Census
17%
complete
12%

Bureau
17%
the
11%

will
12%
American
17%

be
13%
Community
16%

sending
14%
Survey.
16%

you
13%

information
14%

Please
41%
survey
49%

look
43%
as
48%

out
42%
soon
51%

for
41%
as
50%

this
41%
possible.
50%

information
42%

in
41%

the
41%

mail
42%

and
40%

complete
49%

this
46%

important
48%

Federal,
9%
meet
8%
and
7%

tribal,
8%
the
7%
hospitals.
8%

state,
8%
needs
8%

and
7%
of
7%

local
8%
communities
8%

leaders
9%
across
6%

rely
9%
America,
6%

on
8%
including
5%

to
7%
to
7%

understand
7%
roads,
9%

and
6%
schools,
9%

YOUR
78%

RESPONSE
82%

IS
79%

REQUIRED
88%

BY
84%

LAW
87%

The
12%
as
7%

Census
15%
part
7%

Bureau
16%
of
8%

has
14%
a
7%

randomly
22%
nationally
7%

selected
20%
representative
7%

your
18%
sample.
7%

address--not
21%

you
17%

personally--to
17%

receive
9%

the
9%

survey
8%

To
3%
Web
7%

learn
4%
site:
11%

more
4%
https://census...
28%

about
4%

the
4%

American
5%

Community
6%

Survey,
6%

please
5%

visit
5%

the
5%

Census
7%

Bureau’s
6%

Thank
1%

you
1%

in
1%

advance
1%

for
<1%

your
<1%

response.
1%

receive
8%

more
8%

than
10%

$400
28%

billion
28%

toward
13%

Patriotic Pre-notice
Message Highlighting
Word-by-Word Frequency Chart

this
information
7%
8%
bringing improvements
6%
9%

Sincerely,
<1%
John
2%
Director
4%
U.S.
4%

H.
1%

Thompson
2%

Census
5%

Bureau
4%

This
4%

year
4%

communities
7%

across
6%

the
6%

country
7%

will
6%

highway
15%

planning,
15%

medical
14%

assistance,
13%

and
10%

special
14%

education
14%

74

ACS Online Visual Testing
15.

If you had to explain the purpose of the item you just saw to a friend, how would you
describe it to them? (CODED OPEN-ENDS)

Pre-notice letter (Patriotic)
Count (n-size)

491

Letter from the Census Bureau (mentioned Census specifically)
Survey helps determine the needs of my community / How funds are allocated
Mandatory / Required by law
Survey / Reminder to fill out a survey (did not mention Census)
Randomly selected to do the survey
Explaining the importance of filling out the survey/why they are conducting the survey
Letter about previous or soon to come letter / Instructions coming soon
Junk mail / Spam / Garbage / Nonsense
Not legitimate / A scam / Not part of the regular census
Unnecessary / Useless
Government snooping / Using threats / Intimidation

34%
21%
19%
8%
6%
3%
2%
2%
1%
1%
1%
<1%
<1%
2%
1%

More than one way to fill out the survey
All other negatives
Others
Don't Know / No Response

75

ACS Online Visual Testing
16.

Thinking about the item you just read, were there any parts you found to be unclear or
confusing? (CODED OPEN-ENDS)

Pre-notice letter (Patriotic)
Count (n-size)
It was fine / Nothing was unclear
Everything was unclear
How will the funds be distributed / Will the data be used
Why is this required by law / What happens if I do not respond
What is the survey about / Need to see the real survey
How was I 'Randomly selected'
Letter was too vague / Was not sure what the letter was about
Who benefits from my information / Does my information really help
When is it due / How long do we get to complete it
Who is this organization / Never heard of ACS
Addressed to 'resident' / Not addressed to me personally
Cluttered / Too many fonts/sizes/colors / Writing too small to read easily
Others
Don't Know / Did Not Read It All / No Response

76

465
83%
3%
3%
3%
2%
1%
<1%
<1%
<1%
<1%
<1%
<1%
1%
2%

ACS Online Visual Testing

17.

Pre-notice letter (Control)

(INSTRUCTIONS) Please click on the image below in areas that caught your attention when
you first looked at this letter. This could be pictures, words, logos, etc. Please click three times.

(%)
Print Code
Logo
Return 1
Address
Greeting
Paragraph 1
Paragraph 2
Link
Signature
Footer

Click
1st Click

2nd Click

3rd Click

1%
8%
74%
5%
4%
5%
2%
<1%
1%
<1%

1%
15%
23%
25%
11%
12%
7%
3%
1%
1%

3%
13%
15%
18%
14%
13%
11%
6%
4%
3%

Pre2_Control

1st Click

77

ACS Online Visual Testing

2nd Click

3rd Click

78

ACS Online Visual Testing

“Control” Pre-notice Message Highlighting #1
(INSTRUCTIONS) Now looking closer at the content of the letter, please identify the words or
phrases that you find most compelling by highlighting that portion of the text by clicking and
dragging your cursor over the words. You may highlight up to 40 words. After you are finished,
click on the arrow at the bottom of the screen to continue.

79

ACS Online Visual Testing
Control Pre-notice
Message Highlighting
Word-by-Word Frequency Chart

Dear

Resident:

2%

4%

In

a

few

days

your

household

will

receive

13%

14%

19%

19%

19%

22%

22%

25%

how

to

complete

a

very

important

national

survey,

the

18%

19%

23%

46%

the

the

survey

9%

9%

11%

13%

47%
promptly
.
17%

14%

11%

36%
complet
e
13%

45%

follow

26%
instruction
s
11%

conducting

this

survey

and

chose

your

address,

not

you

19%
a
12%

16%
randomly
20%

19%

20%

22%

17%

17%

The

American

about

various

12%

23%
Informatio
n
10%

18%

20%

jobs.
34%
government
s
25%

to

18%
12%
selected sample.
16%
16%

Communit
informatio
Survey collects
y
n
23%
23%
23%
23%

instruction
s
29%

Please

25%

9%

Bureau

is

43%

42%

17%

as

part

of

10%

11%

11%

topics

like

education,

housing,

and

22%

17%

35%

34%

23%

from

this

survey

is

used

by

federal,

state,

local,

and

tribal

8%

14%

14%

10%

15%

26%

26%

25%

15%

23%

across

America.

For

example,

community

leaders

2%

2%

9%

10%

to

meet

the

needs

of

12%

14%

20%

15%

16%
communitie
s
21%

to

decide

where

schools,

10%

16%

17%

29%

13%
15%
highways
hospitals,
,
28%
27%
develop programs

needed.

The

survey

also

is

used

to

21%
provide
19%

3%
job
21%

3%
training,
23%

3%
and
13%

4%
plan
17%

6%
for
14%

6%
the
14%

12%
health
23%

12%
care
23%

If
3%

you
3%

the
5%

Internet
8%

and
4%

want
4%

visit

the

Census
13%

have
access
to
4%
5%
5%
Communit
American
Survey, please
y
11%
11%
11%
7%
advanc
you
in
for
e
1%
<1%
<1%
<1%

8%

7%

your

help.

<1%

1%

Sincerely,
<1%
Thompson
2%

Survey.

Census

9%

John
2%

17%

U.S.

this

1%

on

22%

40%
personall
y,
18%

9%

5%

mail

The

use

Thank

the

20%
Communit
American
y
26%
26%

23%

17%
informatio
n
12%

the

in
21%

Director
3%

Enclosure
2%

80

and

other

services

are

16%

20%

23%

to

reduce

traffic

9%
needs
19%

20%
of
13%

24%
the
13%

19%
congestio
n,
24%
elderly.
19%

to
5%

learn
6%

Bureau’s

Web

site:

12%

15%

15%

more
6%
www.census..
..
32%

about
5%

ACS Online Visual Testing
18.

If you had to explain the purpose of the item you just saw to a friend, how would you
describe it to them? (CODED OPEN-ENDS)

Pre-notice Letter (Control)
Count (n-size)

480

Survey helps determine the needs of my community / How funds are allocated

47%

Letter from the Census Bureau (mentioned Census specifically)

19%

Survey / Reminder to fill out a survey (did not mention Census)

15%

Randomly selected to do the survey

5%

Explaining the importance of filling out the survey/why they are conducting the survey

3%

Letter about previous or soon to come letter / Instructions coming soon

1%

Online survey / Important to complete online

1%

Not legitimate / A scam / Not part of the regular census

1%

Junk mail / Spam / Garbage / Nonsense

1%

Unnecessary / Useless

1%

Government snooping / Using threats / Intimidation

1%

All other negatives

1%

Mandatory / Required by law

<1%

Voluntary
Others

<1%

Don't Know / No Response

2%

1%

81

ACS Online Visual Testing
19.

Thinking about the item you just read, were there any parts you found to be unclear or
confusing? (OPEN-ENDED, CODED)

Pre-notice Letter (Control)
Count (n-size)

459

It was fine / Nothing was unclear

82%

What is the survey about / Need to see the real survey

3%

Everything was unclear

2%

Cluttered / Too many fonts/sizes/colors / Writing too small to read easily

2%

Letter was too vague / Was not sure what the letter was about

1%

Who benefits from my information / Does my information really help

1%

How will the funds be distributed / Will the data be used

1%

Why is this required by law / What happens if I do not respond

1%

Confusion about how to respond / response mode

1%

When is it due / How long do we get to complete it

1%

Who is this organization / Never heard of ACS

1%

Addressed to 'resident' / Not addressed to me personally

1%

Waste of money sending multiple mailings

1%

Is it legitimate / Looks like junk mail

1%

How was I 'Randomly selected'
Others

<1%

Don't Know / Did Not Read It All / No Response

1%

2%

82

ACS Online Visual Testing

Image Click #2 (Internet Invitation Envelope)
(ALL SAW THE FOLLOWING) Now we will have you take a closer look at an envelope.
First, we will ask about your initial visual impression of the envelope. Then we will ask
some follow-up questions.

20.

Internet Invitation Envelope (Official Front)

(INSTRUCTIONS) Please click on the image below in areas that caught your attention when
you first looked at this letter. This could be pictures, words, logos, etc. Please click three times.

(%)
Logo
Stamp
Return 1
Return 2
Address
Open Immediately
Required

Click
1st Click

2nd Click

3rd Click

57%
0%
3%
<1%
2%
11%
27%

28%
0%
2%
0%
5%
17%
49%

22%
0%
6%
0%
16%
38%
18%

A1_C1_Official_Front

1st Click

83

ACS Online Visual Testing

2nd Click

3rd Click

84

ACS Online Visual Testing

21.

Internet Invitation Envelope (Official Back)

(INSTRUCTIONS) Please click on the image below in areas that caught your attention when
you first looked at this letter. This could be pictures, words, logos, etc. Please click three times.

(%)
Required
Logo
Multi-language

Click
st

1 Click

2nd Click

3rd Click

41%
55%
4%

56%
36%
9%

52%
18%
31%

A1_C1_Official_Back

1st Click

2nd Click

85

ACS Online Visual Testing

3rd Click

86

ACS Online Visual Testing

22.

Internet Invitation Envelope (Community Front)

(INSTRUCTIONS) Please click on the image below in areas that caught your attention when
you first looked at this letter. This could be pictures, words, logos, etc. Please click three times.

(%)
Logo
Stamp
Return 1
Return 2
Address
Image (Virginia)
Required
Footer

Click
1st Click

2nd Click

3rd Click

70%
0%
<1%
<1%
5%
4%
19%
1%

25%
1%
2%
1%
9%
6%
54%
1%

26%
0%
10%
1%
26%
9%
23%
4%

A1_C1_Community_Front

1st Click

2nd Click

87

ACS Online Visual Testing

3rd Click

88

ACS Online Visual Testing

23.

Internet Invitation Envelope (Community Back)

(INSTRUCTIONS) Please click on the image below in areas that caught your attention when
you first looked at this letter. This could be pictures, words, logos, etc. Please click three times.

(%)
Logo
Required
Multi-language
Footer

Click
st

1 Click

2nd Click

3rd Click

76%
23%
1%
0%

24%
72%
4%
<1%

22%
40%
25%
13%

A1_C1_Community_Back

1st Click

2nd Click

89

ACS Online Visual Testing

3rd Click

90

ACS Online Visual Testing

24.

Internet Invitation Envelope (Patriotic Front)

(INSTRUCTIONS) Please click on the image below in areas that caught your attention when
you first looked at this letter. This could be pictures, words, logos, etc. Please click three times.

(%)
Logo
Stamp
Return 1
Return 2
Required
Address
Multi-language
Footer

Click
st

1 Click

2nd Click

3rd Click

54%
1%
3%
0%
36%
3%
1%
2%

33%
<1%
6%
2%
42%
13%
1%
3%

10%
1%
11%
3%
18%
36%
5%
15%

A1_C1_Patriotic_Front

1st Click

2nd Click

91

ACS Online Visual Testing

3rd Click

92

ACS Online Visual Testing

25.

Internet Invitation Envelope (Patriotic Back)

(INSTRUCTIONS) Please click on the image below in areas that caught your attention when
you first looked at this letter. This could be pictures, words, logos, etc. Please click three times.
Click

(%)

st

Logo
Required
Image (We the People)
Footer

1 Click

2nd Click

3rd Click

65%
15%
12%
8%

26%
42%
17%
16%

13%
30%
25%
32%

A1_C1_Patriotic_Back

1st Click

2nd Click

93

ACS Online Visual Testing

3rd Click

94

ACS Online Visual Testing

26.

Internet Invitation Envelope (Control Front)

(INSTRUCTIONS) Please click on the image below in areas that caught your attention when
you first looked at this letter. This could be pictures, words, logos, etc. Please click three times.

(%)
Return 1
Equal Opportunity
Stamp
Return 2
Required
Address
Logo

Click
st

1 Click

2nd Click

3rd Click

32%
0%
<1%
1%
57%
7%
2%

39%
<1%
2%
2%
34%
14%
8%

20%
1%
4%
7%
17%
27%
26%

A1_C1_Control_Front

1st Click

2nd Click

95

ACS Online Visual Testing

3rd Click

96

ACS Online Visual Testing

Attributes
Please indicate how much you believe the following words apply to the envelope you
just viewed.
% “Applies completely”

(%)
Official
Important
Attention-grabbing
Easy-to-understand
Urgent
Trustworthy
Informative
27.

Count (n-size)
Applies completely
Applies somewhat
Applies a little
Does not apply at all
Don't know

Community

Patriotic

Control

69
65
60
49
58
30
31

60
53
43
43
30
31
28

64
58
56
51
39
34
31

63
48
44
41
33
28
22

All

Package
Official

Community

Patriotic

Control

2009

509

500

501

499

64
23
8
3
2

69
18
9
2
2

60
26
9
3
2

64
23
9
2
2

63
24
5
3
2

Important
(%)
Count (n-size)
Applies completely
Applies somewhat
Applies a little
Does not apply at all
Don't know

29.

Official

64
56
51
46
40
31
28

Official
(%)

28.

ALL

All

Package
Official

Community

Patriotic

Control

2009

509

500

501

499

56
27
10
4
2

65
21
8
3
2

53
29
12
3
2

58
27
10
3
2

48
33
11
5
2

Attention-grabbing
(%)
Count (n-size)
Applies completely
Applies somewhat
Applies a little
Does not apply at all
Don't know

All

Package
Official

Community Patriotic

Control

2009

509

500

501

499

51
31
12
4
1

60
24
9
4
1

43
37
12
5
1

56
29
10
4
1

44
32
17
4
1

97

ACS Online Visual Testing

30.

Easy-to-understand
(%)
Count (n-size)
Applies completely
Applies somewhat
Applies a little
Does not apply at all
Don't know

31.

Count (n-size)
Applies completely
Applies somewhat
Applies a little
Does not apply at all
Don't know

Community Patriotic

Control

2009

509

500

501

499

46
31
15
5
2

49
29
14
4
2

43
35
15
4
1

51
29
13
5
1

41
32
16
7
2

All

Package
Official

Community

Patriotic

Control

2009

509

500

501

499

40
31
17
8
2

58
25
10
4
2

30
35
22
9
2

39
32
18
7
2

33
32
18
11
3

Official

Community

Patriotic

Control

2009

509

500

501

499

31
32
20
9
6

30
30
22
9
7

31
31
21
9
6

34
35
18
8
4

28
32
21
10
7

Trustworthy
(%)
Count (n-size)
Applies completely
Applies somewhat
Applies a little
Does not apply at all
Don't know

33.

Package
Official

Urgent
(%)

32.

All

All

Package

Informative
(%)
Count (n-size)
Applies completely
Applies somewhat
Applies a little
Does not apply at all
Don't know

All

Package
Official

Community

Patriotic

Control

2009

509

500

501

499

28
35
23
10
2

31
31
22
11
3

28
38
23
8
1

31
35
23
9
2

22
37
25
12
3

98

ACS Online Visual Testing

Image Click #3 (Internet Invitation Letter)
(ALL SAW THE FOLLOWING) Now we will have you take a closer look at a letter that would be
inside the envelope you just saw. First, we will ask about your initial visual impression of the
letter. Then we will ask about the content of the letter.

34.

Internet Invitation Letter
(Official)

(INSTRUCTIONS) Please click on the
image below in areas that caught your
attention when you first looked at this
letter. This could be pictures, words,
logos, etc. Please click three times.

(%)
Logo
Return 1
Greeting
Paragraph 1
Link
Paragraph 2
Paragraph 3
Paragraph 4
Paragraph 5
Paragraph 6
Paragraph 7
Signature
Print Code

A3_Official

1st Click

99

Click
st

1 Click

2nd Click

3rd Click

52%
9%
<1%
1%
32%
1%
<1%
4%
<1%
<1%
0%
<1%
<1%

18%
15%
1%
1%
39%
3%
<1%
21%
2%
<1%
<1%
<1%
0%

8%
9%
1%
3%
17%
2%
1%
41%
14%
1%
1%
1%
0%

ACS Online Visual Testing

2nd Click

3rd Click

100

ACS Online Visual Testing

“Official” Internet Invitation Letter Message Highlighting #1
(INSTRUCTIONS) Now looking closer at the content of the letter, please identify the
words or phrases that you find most compelling by highlighting that portion of the text by
clicking and dragging your cursor over the words. You may highlight up to 40 words.
After you are finished, click on the arrow at the bottom of the screen to continue.

101

ACS Online Visual Testing

U.S.

Department

26%

28%

24%

28%

Economics

and

Statistics

of

Administration
12%

12%

10%

13%

U.S.

Census

Bureau

32%

34%

33%

National

Processing

Center

“Official” Internet Invitation Letter (A_3)
Messaging Highlighting
Word-by-Word Frequency Chart

Commerce

6%

6%

6%

1201

East

10th

Street

Jeffersonville

IN

<1%

<1%

<1%

<1%

1%

1%

1%

An

important

message

from

the

Director,

U.S.

Census

Bureau

8%

13%

13%

9%

9%

13%

14%

15%

15%

Your

address

has

been

randomly

chosen

to

complete

the

8%

9%

9%

10%

17%

15%

7%

7%

7%

conducts

this

survey

each

month

to

give

our

6%

5%

6%

8%

8%

5%

6%

5%

live--our

education,

employment,

housing,

and

more.

8%

9%

9%

9%

5%

6%

Using

the

enclosed

Internet

instruction

card,

please

33%

32%

34%

37%

36%

36%

36%

at

https://respond...

47132

American Community Survey.

The

Census

4%

6%

6%

picture

of

how

we

8%

6%

8%

7%

online

as

soon

as

possible

43%

38%

39%

39%

39%

services

are

needed,

including

things

9%

7%

8%

3%

3%

12%

12%

12%

country

an

up-to-date

5%

6%

8%

complete

the

survey

43%

39%

44%

32%

39%

Communities

use

information

from

this

survey

to

decide

8%

7%

8%

7%

7%

7%

6%

7%

7%

9%

Reducing

traffic

congestion

15%

16%

15%

Providing

job-training

programs
14%

where important

Bureau

like:
3%

14%

17%

Building

schools

and

hospitals

16%

17%

12%

16%

If

you

are

unable

to

complete

the

survey

online,

there

is

no

need

to

contact

3%

3%

3%

4%

3%

3%

3%

4%

4%

3%

3%

4%

4%

3%

4%

us.

We

will

send

you

a

paper

questionnaire

in

a

few

weeks.

193).

3%

3%

3%

4%

4%

4%

5%

5%

3%

3%

3%

3%

Your

response

is

required

by

law

(Title

13,

U.S.

Code,

Sections

141

and

64%

71%

65%

78%

74%

77%

11%

7%

8%

7%

7%

7%

6%

7%

As

a

representative

member

of

your

community,

you

will

be

asked

to

provide

information

3%

3%

3%

3%

3%

3%

3%

2%

2%

2%

2%

2%

3%

3%

2%

your

household

that

is

critical

to

meeting

needs

in

your

area.

Because

you’ll

be

speaking

2%

3%

1%

2%

3%

2%

2%

2%

2%

2%

2%

<1%

1%

1%

2%

on

behalf

of

your

neighbors

and

other

members

of

your

community
,

your

response

is

critical

1%

1%

1%

1%

1%

1%

1%

1%

2%

2%

2%

3%

1%

1%

to

our

ability

to

collect

accurate

data.

<1%

1%

<1%

<1%

1%

1%

1%

1%

Your

responses

will

be

kept

39%

43%

40%

40%

42%

the

Census

Bureau

cannot

1%

2%

2%

2%

may

answers

We

completely confidential.

about

By

law

(Title

13,

U.S.

Code,

Section

9),

48%

53%

5%

5%

3%

3%

3%

3%

3%

2%

publish

or

release

information

that

would

identify

your

household.

2%

2%

2%

2%

2%

2%

2%

2%

2%

with

information

to

other

agencies

combine

your

you

gave

to

enhance

2%

2%

3%

2%

3%

2%

3%

2%

2%

3%

2%

3%

3%

2%

2%

the

statistical

use

of

these

data.

The

information

that

will

be

given

the

same

protections

as

1%

2%

1%

1%

1%

1%

<1%

1%

1%

<1%

1%

<1%

1%

1%

<1%

your

survey

responses.

Based

on

the

answers

you

provide,

you

may

be

asked

to

participate

<1%

1%

1%

2%

1%

1%

2%

2%

2%

2%

3%

3%

4%

2%

4%

in

other

Census

Bureau

surveys

that

are

voluntary.
4%

3%

4%

5%

5%

5%

3%

3%

If

you

need

help

completing

the

survey

or

have

questions,

please

call

1-800…

2%

2%

3%

3%

2%

2%

2%

2%

3%

4%

4%

4%

6%

Thank

you

in

advance

for

your

prompt

response.

1%

1%

<1%

<1%

<1%

<1%

1%

1%

John

H.

Thompson

<1%

<1%

<1%

Director,

U.S.

Census

Bureau

2%

2%

2%

2%

Sincerely,
0%

102

ACS Online Visual Testing
35.

If you had to explain the purpose of the item you just saw to a friend, how would you
describe it to them? (CODED OPEN-ENDS)

Internet Invitation Letter (Official)
Count (n-size)
Survey helps determine the needs of my community / How funds are allocated
Letter from the Census Bureau (mentioned Census specifically)
Mandatory / Required by law
Survey / Reminder to fill out a survey (did not mention Census)
Explaining the importance of filling out the survey/why they are conducting the survey
Online survey / Important to complete online
More than one way to fill out the survey
Randomly selected to do the survey
Government snooping / Using threats / Intimidation
Letter about previous or soon to come letter / Instructions coming soon
Not legitimate / A scam / Not part of the regular census
Junk mail / Spam / Garbage / Nonsense
Unnecessary / Useless
All other negatives
Voluntary
Others
Don't Know / No Response

103

475
30%
15%
12%
7%
6%
6%
5%
4%
2%
1%
1%
1%
1%
1%
<1%
2%
4%

ACS Online Visual Testing
35.

Thinking about the item you just read, were there any parts you found to be unclear or
confusing? (CODED OPEN-ENDS)

Internet Invitation Letter (Official)
Count (n-size)
It was fine / Nothing was unclear
Why is this required by law / What happens if I do not respond
Everything was unclear
Who are the other agencies the information will be combined with
What is the survey about / Need to see the real survey
Confusion about how to respond / response mode
Who is this organization / Never heard of ACS
Is it legitimate / Looks like junk mail
Cluttered / Too many fonts/sizes/colors / Writing too small to read easily
Letter was too vague / Was not sure what the letter was about
Who benefits from my information / Does my information really help
How will the funds be distributed / Will the data be used
How was I 'Randomly selected'
When is it due / How long do we get to complete it
Waste of money sending multiple mailings
Others
Don't Know / Did Not Read It All / No Response

104

450
84%
3%
2%
2%
1%
1%
1%
1%
1%
<1%
<1%
<1%
<1%
<1%
<1%
2%
1%

ACS Online Visual Testing

36.

Internet Invitation Letter (Community)

(INSTRUCTIONS) Please click on the image below in areas that caught your attention when
you first looked at this letter. This could be pictures, words, logos, etc. Please click three times.

(%)
Logo
Print Code
Greeting
Paragraph 1
Paragraph 2
Link
Paragraph 3
Paragraph 4
Paragraph 5
Paragraph 6
Signature
Footer

Click
1st Click

2nd Click

3rd Click

73%
<1%
1%
2%
3%
14%
1%
4%
1%
0%
<1%
<1%

29%
<1%
2%
3%
5%
35%
4%
15%
3%
1%
<1%
2%

12%
<1%
3%
3%
6%
22%
4%
30%
11%
2%
3%
4%

A3_Community

1st Click

105

ACS Online Visual Testing

2nd Click

3rd Click

106

ACS Online Visual Testing

“Community” Internet Invitation Letter Message Highlighting #1
(INSTRUCTIONS) Now looking closer at the content of the letter, please identify the words or
phrases that you find most compelling by highlighting that portion of the text by clicking and
dragging your cursor over the words. You may highlight up to 40 words. After you are finished,
click on the arrow at the bottom of the screen to continue.

107

ACS Online Visual Testing
U.S.
31%
Washington,
8%

Census
32%
DC
6%

Bureau
31%
20233
5%

A
7%

message
10%

from
9%

Your
13%
American
14%
number
4%
nation-including
6%
Communities
12%

“Community” Internet Invitation Letter (A_3)
Message Highlighting
Word-by-Word Frequency Chart
the
9%

U.S.
13%

Census
14%

address
has
been randomly selected
14%
12%
13%
22%
20%
Community Survey.
The
American Community
13%
13%
2%
3%
3%
of
people
who
live
in
4%
4%
3%
4%
3%
our
education, housing,
and
jobs.
4%
8%
8%
5%
7%

Bureau:
13%
by
11%
Survey
3%
the
3%

the
12%
and
2%
country,
4%

U.S.
14%
the
3%
but
2%

to
12%

decide
14%

where
13%

Census
14%
Census
4%
also
2%

Bureau
13%
show
4%
how
5%

to
8%
us
3%
we
6%

participate
9%
not
3%
live
6%

in
8%
only
3%
as
4%

important services
14%
14%

are
13%

needed,
14%

including:
7%

the
8%
the
3%
a
4%

use
12%

information
14%

from
11%

this
12%

survey
12%

Improving
24%
Building
21%
Planning
20%

roads
23%
schools
21%
for
15%

and
15%

reducing
21%

traffic
21%

the
14%

health
21%

care
20%

needs
18%

of
13%

the
12%

elderly
18%

Using
28%
https://respond....
40%

the
27%

enclosed
30%

card,
30%

please
31%

complete
40%

the
36%

survey
40%

online
42%

as
36%

soon
37%

as
36%

possible
36%

at:
31%

If
13%
us.
15%
is
1%
save
2%

you
12%
We
3%
using
2%
taxpayers’
2%

are
12%
will
3%
the
2%
money,
2%

unable
16%
send
4%
Internet
2%
and
1%

to
14%
you
4%
to
1%
process
1%

complete
16%
a
3%
collect
1%
your
1%

the
14%
paper
5%
this
1%
data
1%

there
14%
a
1%
an
1%

is
14%
few
3%
effort
2%

no
17%
weeks.
3%
to
1%

need
17%
The
1%
conserve
3%

to
16%
Census
1%
natural
3%

contact
17%
Bureau
2%
resources,
3%

Your
67%
Because
5%
that
5%

response
71%
you
5%
you
5%

is
required
68%
80%
have
been
5%
5%
complete
this
6%
5%

by
75%
selected
5%
survey.
5%

law
78%
to
4%

(Title
13%
participate
5%

and
9%
it
5%

193).
9%
is
5%

vital
7%

Your
27%
the
1%
We
<1%
the
<1%
your
<1%
participate
1%

response
30%
Census
2%
may
1%
statistical
<1%
survey
<1%
in
1%

will
27%
Bureau
1%
combine
<1%
uses
<1%
response.
<1%
other
1%

be
27%
cannot
2%
your
<1%
of
<1%
Based
<1%
Census
1%

kept
31%
publish
2%
answers
<1%
these
<1%
on
0%
Bureau
1%

If
2%

you
2%

need
3%

help
4%

completing
3%

the
2%

survey
2%

or
2%

Thank
1%

you
1%

in
<1%

advance
<1%

for
<1%

your
1%

prompt
2%

response.
2%

survey
online,
16%
16%
questionnaire
in
5%
2%
information
in
1%
1%
more
efficiently.
1%
2%

H.
<1%
U.S.
2%

U.S.
8%
behalf
5%

Code,
9%
of
5%

Sections
141
8%
9%
your community,
5%
5%

completely confidential.
By
law
(Title
13,
36%
41%
4%
3%
2%
2%
or
release
information
that
would identify
1%
2%
2%
1%
1%
2%
with
information
that
you
gave
to
<1%
<1%
<1%
<1%
<1%
<1%
data.
This
information
will
be
given
<1%
<1%
<1%
0%
0%
<1%
the
information
that
you
provide,
you
0%
<1%
0%
<1%
<1%
<1%
surveys
that
are
voluntary.
1%
1%
1%
1%

Sincerely,
<1%
John
1%
Director,
2%

13,
9%
on
4%

Thompson
1%
Census
Bureau
2%
2%

108

have
2%

questions, please
3%
4%

U.S.
2%
you
2%
other
<1%
the
0%
may
<1%

call
7%

Code,
Section
9),
2%
1%
1%
or
your
household.
1%
1%
1%
agencies
to
enhance
<1%
<1%
<1%
same
protection
as
<1%
<1%
<1%
be
asked
to
<1%
1%
<1%

1-800…
0%

ACS Online Visual Testing
37.

If you had to explain the purpose of the item you just saw to a friend, how would you
describe it to them? (CODED OPEN-ENDS)

Internet Invitation Letter (Community)
Count (n-size)
Survey helps determine the needs of my community / How funds are allocated
Letter from the Census Bureau (mentioned Census specifically)
Mandatory / Required by law
Online survey / Important to complete online
More than one way to fill out the survey
Survey / Reminder to fill out a survey (did not mention Census)
Explaining the importance of filling out the survey/why they are conducting the survey
Randomly selected to do the survey
Letter about previous or soon to come letter / Instructions coming soon
Not legitimate / A scam / Not part of the regular census
Junk mail / Spam / Garbage / Nonsense
Government snooping / Using threats / Intimidation
Unnecessary / Useless
All other negatives
Voluntary
Others
Don't Know / No Response

38.

463
33%
16%
9%
9%
8%
5%
4%
3%
2%
2%
2%
2%
1%
1%
<1%
2%
3%

Thinking about the item you just read, were there any parts you found to be unclear or
confusing? (CODED OPEN-ENDS)

Internet Invitation Letter (Community)
Count (n-size)
It was fine / Nothing was unclear
Why is this required by law / What happens if I do not respond
Everything was unclear
What is the survey about / Need to see the real survey
Who benefits from my information / Does my information really help
How was I 'Randomly selected'
Is it legitimate / Looks like junk mail
Cluttered / Too many fonts/sizes/colors / Writing too small to read easily
Letter was too vague / Was not sure what the letter was about
Confusion about how to respond / response mode
Who is this organization / Never heard of ACS
Others
Don't Know / Did Not Read It All / No Response

109

434
87%
3%
2%
1%
1%
1%
1%
1%
<1%
<1%
<1%
<1%
1%

ACS Online Visual Testing

39.

Internet Invitation Letter (Patriotic)

(INSTRUCTIONS) Please click on the image below in areas that caught your attention when
you first looked at this letter. This could be pictures, words, logos, etc. Please click three times.
Click

(%)

st

Logo
Header (We The People)
Greeting
Paragraph 1
Link
Paragraph 2
Paragraph 3
Paragraph 4
Paragraph 5
Signature
Print Code
Paragraph 6
Footer

1 Click

2nd Click

3rd Click

62%
1%
<1%
5%
21%
4%
3%
1%
0%
0%
0%
2%
1%

13%
2%
<1%
11%
46%
5%
13%
3%
0%
<1%
0%
5%
2%

7%
2%
<1%
11%
12%
5%
21%
11%
3%
3%
0%
20%
4%

A3_Patriotic

1st Click

110

ACS Online Visual Testing

2nd Click

3rd Click

111

ACS Online Visual Testing

“Patriotic” Internet Invitation Letter Message Highlighting #1
(INSTRUCTIONS) Now looking closer at the content of the letter, please identify the words or
phrases that you find most compelling by highlighting that portion of the text by clicking and
dragging your cursor over the words. You may highlight up to 40 words. After you are finished,
click on the arrow at the bottom of the screen to continue.

112

ACS Online Visual Testing
Dear
1%

Resident,
2%

Enclosed
7%
to
17%

you
5%
the
16%

“Patriotic” Internet Invitation Letter (A_3)
Message Highlighting
Word-by-Word Frequency Chart
will
5%
health
20%

Businesses
create
0%
17%
Nonprofit organizations
19%
0%
Federal,
tribal,
10%
11%

find
7%
of
18%

instructions
9%
our
18%

jobs
19%
serve
14%
state,
0%

their
14%
and
14%

for
7%
nation.
20%

completing
8%
Data
10%

the
9%
collected
10%

populations
12%
local
governments
14%
14%

American Community Survey.
35%
34%
0%
help:
9%

This
12%

survey
15%

meet
11%

the
15%

needs
15%

of
13%

their
12%

communities
15%

is
14%

vital
20%

Responding
12%
unable
6%
send
5%

online
12%
to
5%
you
5%

is
14%
complete
5%
a
5%

the
56%
the
5%
paper
5%

quickest
59%
survey
5%
questionnaire.
5%

and
49%
online,
5%

easiest
48%
there
5%

way
54%
is
5%

to
47%
no
5%

complete
51%
need
5%

the
47%
to
5%

survey.
44%
contact
5%

If
47%
us.
5%

you
44%
We
5%

are
45%
will
5%

Responding
5%
193).
9%
duty.
7%

to
6%
Like
9%

this
6%
responding
9%

survey
67%
for
10%

is
63%
jury
8%

required
63%
duty,
8%

by
68%
completing
8%

law
66%
this
9%

(Title
81%
Census
9%

13,
79%
Bureau
7%

U.S.
81%
survey
6%

Code,
15%
is
7%

Sections
11%
an
7%

141
10%
important
7%

and
9%
civic
6%

(Title
47%
gave
2%
be
<1%
you
1%
Bureau
2%

13,
44%
to
2%
given
<1%
may
1%
employees
2%

U.S.
53%
other
2%
the
<1%
be
1%
are
1%

help
4%

completing
4%

the
4%

survey
5%

or
4%

have
4%

data
8%

help
8%

determine
8%

the
5%

annual
6%

distribution
6%

All
answers
are
confidential
9%
8%
8%
43%
with
information
that
you
2%
2%
2%
2%
data.
This
information
will
1%
1%
1%
1%
the
answers
you
provide,
1%
1%
1%
1%
that
are
voluntary.
Census
3%
2%
2%
2%
information.
1%
If
2%

you
2%

need
2%

Code,
Section
9).
6%
4%
4%
agencies
to
enhance
2%
2%
2%
same
protections
as
<1%
<1%
<1%
asked
to
participate
2%
2%
2%
prohibited
by
law
1%
1%
1%

We
may
combine
your
answers
4%
3%
2%
2%
2%
the statistical
use
of
these
3%
1%
1%
1%
1%
your
survey
responses.
Based
on
<1%
<1%
<1%
<1%
1%
in
other
Census
Bureau
surveys
2%
2%
2%
2%
2%
from releasing
any
personally identifiable
2%
1%
1%
2%
2%

questions, please
5%
4%

call
4%

1-800…
4%

than
8%

$400
7%

Sincerely,
6%
John
8%
Director
1%
U.S.
<1%

H.
17%

Thompson
<1%

Census
1%

Bureau
1%

American
2%
federal
23%

Community
2%
funds.
22%

Survey
2%

113

of
5%

more
7%

billion
8%

in
9%

ACS Online Visual Testing

40.

If you had to explain the purpose of the item you just saw to a friend, how would
you describe it to them? (CODED OPEN-ENDS)

Internet Invitation Letter (Patriotic)
Count (n-size)
Survey helps determine the needs of my community / How funds are allocated
Letter from the Census Bureau (mentioned Census specifically)
Mandatory / Required by law
More than one way to fill out the survey
Survey / Reminder to fill out a survey (did not mention Census)
Online survey / Important to complete online
Explaining the importance of filling out the survey/why they are conducting the survey
Letter about previous or soon to come letter / Instructions coming soon
Government snooping / Using threats / Intimidation
Not legitimate / A scam / Not part of the regular census
Junk mail / Spam / Garbage / Nonsense
All other negatives
Randomly selected to do the survey
Voluntary
Unnecessary / Useless
Others
Don't Know / No Response

114

471
27%
16%
14%
8%
7%
7%
6%
3%
2%
1%
1%
1%
<1%
<1%
<1%
3%
4%

ACS Online Visual Testing

41.

Thinking about the item you just read, were there any parts you found to be
unclear or confusing? (CODED OPEN-ENDS)

Internet Invitation Letter (Patriotic)
Count (n-size)

456

It was fine / Nothing was unclear
Why is this required by law / What happens if I do not respond
Everything was unclear
What is the survey about / Need to see the real survey
How will the funds be distributed / Will the data be used
Who are the other agencies the information will be combined with
Confusion about how to respond / response mode
Who is this organization / Never heard of ACS
Letter was too vague / Was not sure what the letter was about
Who benefits from my information / Does my information really help
When is it due / How long do we get to complete it
Waste of money sending multiple mailings
Is it legitimate / Looks like junk mail
Cluttered / Too many fonts/sizes/colors / Writing too small to read easily
Others
Don't Know / Did Not Read It All / No Response

81%
4%
3%
2%
2%
2%
1%
1%
<1%
<1%
<1%
<1%
<1%
<1%
2%
1%

115

ACS Online Visual Testing

42.

Internet Invitation Letter (Control)

(INSTRUCTIONS) Please click on the image below in areas that caught your attention when
you first looked at this letter. This could be pictures, words, logos, etc. Please click three times.

(%)
Logo
Return 1
Greeting
Link
Paragraph 1
Paragraph 2
Paragraph 3
Paragraph 4
Signature
Footer

Click
st

1 Click

2nd Click

3rd Click

17%
63%
2%
11%
3%
1%
2%
0%
1%
<1%

13%
24%
12%
29%
7%
5%
6%
1%
1%
0%

8%
10%
14%
26%
10%
10%
13%
5%
1%
1%

A3_Control

1st Click

116

ACS Online Visual Testing

2nd Click

3rd Click

117

ACS Online Visual Testing

“Control” Internet Invitation Letter Message Highlighting #1
(INSTRUCTIONS) Now looking closer at the content of the letter, please identify the words or
phrases that you find most compelling by highlighting that portion of the text by clicking and
dragging your cursor over the words. You may highlight up to 40 words. After you are finished,
click on the arrow at the bottom of the screen to continue.

118

ACS Online Visual Testing

A
8%

message
10%

from
10%

The
13%
Survey.
20%

U.S.
21%
Using
8%

Census
23%
the
8%

Census
17%
natural
33%
to
7%
you
8%

Bureau
17%
resources,
34%
complete
8%
a
8%

the
10%

Director,
19%

“Control” Internet Invitation Letter
Message Highlighting
Word-by-Word Frequency Chart

U.S.
20%

Census
21%

Bureau
21%

…
8%

sent
14%
please
18%

a
13%
complete
24%

letter
14%
the
21%

to
12%
survey
26%

your
12%
online
26%

household
13%
as
24%

using
19%
taxpayers’
34%
survey
8%
questionnaire
12%

the
17%
money,
36%
online,
8%
in
8%

Internet
24%
and
14%
there
6%
a
8%

to
15%
process
22%
is
6%
few
10%

collect
21%
your
18%
no
8%
weeks.
9%

this
17%
data
22%
need
8%

information
in
21%
12%
more
efficiently.
19%
22%
to
contact
8%
8%

up-to-date
23%
results
5%
are
13%
might
5%

information
24%
from
4%
needed.
16%
affect
6%

used
12%
the
4%
This
2%
you
6%

to
11%
survey
4%
information
4%
and
3%

meet
16%
are
4%
also
3%
your
4%

Bureau
recently
24%
15%
enclosed instructions,
9%
9%

about
9%
soon
25%

the
10%
as
24%

American Community
20%
21%
possible
at:
25%
15%

https://respond....
46%
The
9%
conserve
32%
unable
9%
send
10%
This
6%
United
15%
hospitals,
25%
of
8%
natural
9%

is
12%
save
35%
the
7%
paper
12%

survey
collects
critical
11%
17%
26%
States.
For
example,
14%
2%
2%
and
fire
stations
14%
24%
25%
emergency situations
that
22%
20%
5%
disasters.
9%

The
8%
sample.
14%
is
10%
asked
2%

Census
13%
You
27%
required
15%
questions
2%

Bureau
13%
are
27%
by
13%
about
1%

chose
16%
required
46%
U.S.
15%
the
1%

your
16%
by
40%
law
15%
survey.
1%

address,
18%
U.S.
43%
to
12%

not
15%
law
45%
keep
15%

you
15%
to
31%
your
14%

If
1%

you
2%

need
3%

help
3%

completing
2%

the
2%

survey,
2%

please
3%

Thank
1%

you.
1%

John
1%

Thompson
1%

Director
1%
Census
1%

Bureau
1%

Enclosures
1%

119

effort
15%
you
5%
We
8%

to
16%
are
6%
will
9%

of
14%
decide
11%
plan
12%
as
3%

communities
22%
where
10%
for
8%
floods
9%

across
12%
new
18%
the
7%
and
3%

the
10%
schools,
24%
kinds
8%
other
4%

part
8%
this
28%
The
2%

of
8%
survey.
29%
enclosed
3%

a
8%
The
9%
brochure
3%

toll-free
4%

number
5%

(1-800…
11%

the
needs
13%
20%
used
to
7%
6%
helps
communities
5%
7%
neighbors,
such
7%
3%

personally,
as
15%
7%
respond
to
35%
28%
answers confidential.
18%
20%

call
4%

an
12%
If
5%
us.
8%

our
3%

randomly selected
16%
13%
Census
Bureau
11%
12%
answers frequently
2%
2%

ACS Online Visual Testing

43.

If you had to explain the purpose of the item you just saw to a friend, how would
you describe it to them? (CODED OPEN-ENDS)

Internet Invitation Letter (Control)
Count (n-size)
Survey helps determine the needs of my community / How funds are allocated
Online survey / Important to complete online
Letter from the Census Bureau (mentioned Census specifically)
Survey / Reminder to fill out a survey (did not mention Census)
Mandatory / Required by law
More than one way to fill out the survey
Randomly selected to do the survey
Government snooping / Using threats / Intimidation
Explaining the importance of filling out the survey/why they are conducting the survey
Letter about previous or soon to come letter / Instructions coming soon
Not legitimate / A scam / Not part of the regular census
Junk mail / Spam / Garbage / Nonsense
Unnecessary / Useless
All other negatives
Voluntary
Others
Don't Know / No Response

120

470
39%
11%
10%
10%
9%
6%
3%
3%
2%
1%
1%
1%
1%
1%
<1%
<1%
2%

ACS Online Visual Testing

44.

Thinking about the item you just read, were there any parts you found to be
unclear or confusing? (CODED OPEN-ENDS)

Internet Invitation Letter (Control)
Count (n-size)
It was fine / Nothing was unclear
Why is this required by law / What happens if I do not respond
Everything was unclear
Confusion about how to respond / response mode
When is it due / How long do we get to complete it
Who is this organization / Never heard of ACS
Addressed to 'resident' / Not addressed to me personally
Is it legitimate / Looks like junk mail
Cluttered / Too many fonts/sizes/colors / Writing too small to read easily
Letter was too vague / Was not sure what the letter was about
What is the survey about / Need to see the real survey
Who benefits from my information / Does my information really help
How was I 'Randomly selected'
Who are the other agencies the information will be combined with
Waste of money sending multiple mailings
Others
Don't Know / Did Not Read It All / No Response

121

442
85%
5%
2%
1%
1%
1%
1%
1%
1%
<1%
<1%
<1%
<1%
<1%
<1%
1%
1%

ACS Online Visual Testing

Image Click #4 (Internet Instruction Card)
(ALL SAW THE FOLLOWING) Now we will have you take a look at an additional mail item. We
will ask you about your initial visual impression of the mail item. Then we will ask some followup questions.

45.

Internet Instruction Card (Official)

(INSTRUCTIONS) Please click on the image below in areas that caught your attention when
you first looked at this letter. This could be pictures, words, logos, etc. Please click three times.

(%)
Logo
Header
Paragraph 1
Address
Respond now
Multi-language
Link
Required

Click
st

1 Click

2nd Click

3rd Click

44%
2%
17%
3%
26%
0%
2%
6%

20%
6%
20%
3%
17%
0%
12%
21%

9%
9%
12%
5%
17%
0%
16%
33%

A2_Official_Front

1st Click

2nd Click

122

ACS Online Visual Testing

3rd Click

46.

If you had to explain the purpose of the item you just saw to a friend, how would
you describe it to them? (CODED OPEN-ENDS)

Internet Instruction Card (Official)
Count (n-size)

46

Letter from the Census Bureau (mentioned Census specifically)
Online survey / Important to complete online
Mandatory / Required by law
More than one way to fill out the survey
Letter about previous or soon to come letter / Instructions coming soon
Survey / Reminder to fill out a survey (did not mention Census)
Explaining the importance of filling out the survey/why they are conducting the survey
Survey helps determine the needs of my community / How funds are allocated
Government snooping / Using threats / Intimidation
Others
Don't Know / No Response

47.

33%
24%
13%
7%
4%
2%
2%
2%
2%
4%
7%

Thinking about the item you just read, were there any parts you found to be
unclear or confusing? (CODED OPEN-ENDS)

Internet Instruction Card (Official)
Count (n-size)
It was fine / Nothing was unclear
Everything was unclear
Letter was too vague / Was not sure what the letter was about
What is the survey about / Need to see the real survey
Why is this required by law / What happens if I do not respond
Confusion about how to respond / response mode
Is it legitimate / Looks like junk mail
Cluttered / Too many fonts/sizes/colors / Writing too small to read easily
Others
Don't Know / Did Not Read It All / No Response

123

54
85%
2%
2%
2%
2%
2%
2%
2%
2%
0%

ACS Online Visual Testing

48.

Mail Instruction Card (Official)

(INSTRUCTIONS) Please click on the image below in areas that caught your attention when
you first looked at this letter. This could be pictures, words, logos, etc. Please click three times.

(%)
Logo
Header
Paragraph 1
Required
Option 1
Paragraph 2
Option 2
Multi-language

Click
1st Click

2nd Click

3rd Click

57%
2%
1%
28%
10%
0%
1%
0%

20%
3%
5%
39%
23%
1%
8%
1%

19%
11%
12%
12%
19%
4%
21%
1%

C2_Official_Front

1st Click

2nd Click

124

ACS Online Visual Testing

3rd Click

49.

If you had to explain the purpose of the item you just saw to a friend, how would
you describe it to them? (CODED OPEN-ENDS)

Mail Instruction Card (Official)
Count (n-size)
More than one way to fill out the survey
Letter from the Census Bureau (mentioned Census specifically)
Mandatory / Required by law
Survey / Reminder to fill out a survey (did not mention Census)
Randomly selected to do the survey
Online survey / Important to complete online
Survey helps determine the needs of my community / How funds are allocated
Government snooping / Using threats / Intimidation
Others
Don't Know / No Response

50.

54
31%
28%
13%
7%
6%
2%
2%
2%
2%
7%

Thinking about the item you just read, were there any parts you found to be
unclear or confusing? (CODED OPEN-ENDS)

Internet Instruction Card (Official)
Count (n-size)
It was fine / Nothing was unclear
Why is this required by law / What happens if I do not respond
Confusion about how to respond / response mode
Others
Don't Know / Did Not Read It All / No Response

125

63

94%
2%
2%
0%
3%

ACS Online Visual Testing

51.

Reminder Postcard (Official Front)

(INSTRUCTIONS) Please click on the image below in areas that caught your attention when
you first looked at this letter. This could be pictures, words, logos, etc. Please click three times.

(%)
Logo
Stamp
Return 1
Link
Return 2
Multi-language
Address
Print Code

52.

Click
st

1 Click

2nd Click

3rd Click

60%
0%
4%
35%
0%
1%
0%
0%

30%
0%
9%
46%
5%
6%
4%
0%

21%
1%
21%
15%
9%
11%
22%
0%

Reminder Postcard (Official Back)

(%)
Link
Paragraph 1
Required
Paragraph 2
Paragraph 3
Paragraph 4

Click
st

1 Click

2nd Click

3rd Click

63%
2%
33%
1%
2%
0%

27%
5%
51%
3%
13%
1%

13%
9%
11%
19%
40%
8%

(See next pages for heat maps)

126

ACS Online Visual Testing
B1_Official_Front

1st Click

2nd Click

3rd Click

127

ACS Online Visual Testing
B1_Official_Back

1st Click

2nd Click

3rd Click

128

ACS Online Visual Testing

53.

If you had to explain the purpose of the item you just saw to a friend, how would
you describe it to them? (CODED OPEN-ENDS)

Reminder Postcard (Official)
Count (n-size)

123

Letter from the Census Bureau (mentioned Census specifically)
Mandatory / Required by law
Survey / Reminder to fill out a survey (did not mention Census)
Survey helps determine the needs of my community / How funds are allocated
Explaining the importance of filling out the survey/why they are conducting the survey
Online survey / Important to complete online
Letter about previous or soon to come letter / Instructions coming soon
Junk mail / Spam / Garbage / Nonsense
More than one way to fill out the survey
Not legitimate / A scam / Not part of the regular census
Government snooping / Using threats / Intimidation
All other negatives
Randomly selected to do the survey
Others
Don't Know / No Response

36%
12%
9%
9%
7%
7%
6%
4%
2%
2%
2%
2%
1%

54.

2%
2%

Thinking about the item you just read, were there any parts you found to be
unclear or confusing? (CODED OPEN-ENDS)

Reminder Postcard (Official)
Count (n-size)
It was fine / Nothing was unclear
Why is this required by law / What happens if I do not respond
Why is Spanish-language materials included
Is it legitimate / Looks like junk mail
Everything was unclear
Letter was too vague / Was not sure what the letter was about
Cluttered / Too many fonts/sizes/colors / Writing too small to read easily
Others
Don't Know / Did Not Read It All / No Response

129

126
85%
6%
3%
2%
1%
1%
1%
2%
0%

ACS Online Visual Testing

55.

Internet Instruction Card (Community)

(INSTRUCTIONS) Please click on the image below in areas that caught your attention when
you first looked at this letter. This could be pictures, words, logos, etc. Please click three times.

(%)
Logo
Link
Paragraph 1
Address
Keep this card
Multi-language
Footer

Click
st

1 Click

2nd Click

3rd Click

51%
14%
2%
4%
30%
0%
0%

29%
29%
8%
8%
26%
0%
0%

17%
31%
15%
10%
27%
0%
0%

(See next page for heat maps)

130

ACS Online Visual Testing
A2_Community_Front

1st Click

2nd Click

3rd Click

131

ACS Online Visual Testing

56.

If you had to explain the purpose of the item you just saw to a friend, how would
you describe it to them? (CODED OPEN-ENDS)

Internet Instruction Card (Community)
Count (n-size)
Online survey / Important to complete online
Letter from the Census Bureau (mentioned Census specifically)
Survey / Reminder to fill out a survey (did not mention Census)
Letter about previous or soon to come letter / Instructions coming soon
Explaining the importance of filling out the survey/why they are conducting the survey
Mandatory / Required by law
More than one way to fill out the survey
All other negatives
Survey helps determine the needs of my community / How funds are allocated
Junk mail / Spam / Garbage / Nonsense
Others
Don't Know / No Response

57.

61
33%
15%
13%
7%
5%
3%
3%
3%
2%
2%
7%
8%

Thinking about the item you just read, were there any parts you found to be
unclear or confusing? (CODED OPEN-ENDS)

Internet Instruction Card (Community)
Count (n-size)
It was fine / Nothing was unclear
Letter was too vague / Was not sure what the letter was about
Confusion about how to respond / response mode
When is it due / How long do we get to complete it
Addressed to 'resident' / Not addressed to me personally
Cluttered / Too many fonts/sizes/colors / Writing too small to read easily
Others
Don't Know / Did Not Read It All / No Response

132

53
85%
4%
2%
2%
2%
2%
4%
0%

ACS Online Visual Testing

58.

Mail Instruction Card (Community)

(INSTRUCTIONS) Please click on the image below in areas that caught your attention when
you first looked at this letter. This could be pictures, words, logos, etc. Please click three times.

(%)
Logo
Header
Paragraph 1
Option 1
Option 2
Paragraph 2
Print Code
Multi-language

Click
st

1 Click

2nd Click

3rd Click

60%
0%
8%
19%
8%
1%
0%
4%

21%
1%
3%
51%
13%
2%
0%
6%

25%
2%
7%
33%
12%
5%
2%
14%

(See next page for heat maps)

133

ACS Online Visual Testing
C2_ Community _Front

1st Click

2nd Click

3rd Click

134

ACS Online Visual Testing

59.

If you had to explain the purpose of the item you just saw to a friend, how would
you describe it to them? (CODED OPEN-ENDS)

Mail Instruction Card (Community)
Count (n-size)
Letter from the Census Bureau (mentioned Census specifically)
More than one way to fill out the survey
Survey helps determine the needs of my community / How funds are allocated
Mandatory / Required by law
Survey / Reminder to fill out a survey (did not mention Census)
Randomly selected to do the survey
All other negatives
Letter about previous or soon to come letter / Instructions coming soon
Online survey / Important to complete online
Not legitimate / A scam / Not part of the regular census
Government snooping / Using threats / Intimidation
Others
Don't Know / No Response

60.

43
23%
19%
16%
9%
5%
5%
5%
2%
2%
2%
2%
2%
7%

Thinking about the item you just read, were there any parts you found to be
unclear or confusing? (CODED OPEN-ENDS)

Mail Instruction Card (Community)
Count (n-size)
It was fine / Nothing was unclear
Confusion about how to respond / response mode
Cluttered / Too many fonts/sizes/colors / Writing too small to read easily
Everything was unclear
Letter was too vague / Was not sure what the letter was about
Others

Don't Know / Did Not Read It All / No Response

135

52
87%
4%
4%
2%
2%
2%
0%

ACS Online Visual Testing

61.

Reminder Postcard (Community Front)

(INSTRUCTIONS) Please click on the image below in areas that caught your attention when
you first looked at this letter. This could be pictures, words, logos, etc. Please click three times.

(%)
Logo
Stamp
Return 1
Return 2
Address
Required
Paragraph 1

62.

Click
st

1 Click

2nd Click

3rd Click

63%
0%
2%
0%
5%
30%
1%

36%
0%
4%
3%
6%
48%
3%

21%
1%
5%
8%
21%
20%
24%

Reminder Postcard (Community Back)
(%)
Logo
Paragraph 1
Link
Paragraph 2
Paragraph 3
Paragraph 4

Click
1st Click

2nd Click

3rd Click

67%
5%
25%
3%
0%
1%

18%
10%
53%
17%
2%
1%

9%
12%
21%
48%
5%
5%

(See next pages for heat maps)

136

ACS Online Visual Testing
B1_ Community _Front

1st Click

2nd Click

3rd Click

137

ACS Online Visual Testing
B1_ Community _Back

1st Click

2nd Click

3rd Click

138

ACS Online Visual Testing

63.

If you had to explain the purpose of the item you just saw to a friend, how would
you describe it to them? (CODED OPEN-ENDS)

Reminder Postcard (Community)
Count (n-size)
Letter from the Census Bureau (mentioned Census specifically)
More than one way to fill out the survey
Survey / Reminder to fill out a survey (did not mention Census)
Mandatory / Required by law
Online survey / Important to complete online
Survey helps determine the needs of my community / How funds are allocated
Letter about previous or soon to come letter / Instructions coming soon
Explaining the importance of filling out the survey/why they are conducting the survey
Junk mail / Spam / Garbage / Nonsense
Voluntary
Unnecessary / Useless
All other negatives
Others
Don't Know / No Response

64.

120
22%
14%
12%
11%
10%
9%
5%
3%
2%
1%
1%
1%
2%
7%

Thinking about the item you just read, were there any parts you found to be
unclear or confusing? (CODED OPEN-ENDS)

Reminder Postcard (Community)
Count (n-size)
It was fine / Nothing was unclear
Everything was unclear
Who benefits from my information / Does my information really help
How was I 'Randomly selected'
Why is this required by law / What happens if I do not respond
Is it legitimate / Looks like junk mail
Cluttered / Too many fonts/sizes/colors / Writing too small to read easily
Others

Don't Know / Did Not Read It All / No Response

139

120
92%
2%
2%
2%
2%
1%
1%
0%
0%

ACS Online Visual Testing

65.

Internet Instruction Card (Patriotic)

(INSTRUCTIONS) Please click on the image below in areas that caught your attention when
you first looked at this letter. This could be pictures, words, logos, etc. Please click three times.

(%)
Logo
Header
Paragraph 1
Address
Keep this card
Multi-language
Print Code
Footer

Click
st

1 Click

2nd Click

3rd Click

42%
22%
3%
5%
26%
0%
0%
2%

18%
32%
11%
9%
27%
0%
0%
2%

17%
24%
21%
7%
24%
0%
0%
7%

A2_Patriotic_Front

1st Click

140

ACS Online Visual Testing

2nd Click

3rd Click

141

ACS Online Visual Testing

66.

If you had to explain the purpose of the item you just saw to a friend, how would
you describe it to them? (CODED OPEN-ENDS)

Internet Instruction Card (Patriotic)
Count (n-size)
Letter from the Census Bureau (mentioned Census specifically)
Online survey / Important to complete online
Survey / Reminder to fill out a survey (did not mention Census)
Explaining the importance of filling out the survey/why they are conducting the survey
Letter about previous or soon to come letter / Instructions coming soon
Mandatory / Required by law
More than one way to fill out the survey
Junk mail / Spam / Garbage / Nonsense
Others
Don't Know / No Response

67.

47
34%
34%
9%
6%
2%
2%
2%
2%
4%
4%

Thinking about the item you just read, were there any parts you found to be
unclear or confusing? (CODED OPEN-ENDS)

Internet Instruction Card (Patriotic)
Count (n-size)
It was fine / Nothing was unclear
Everything was unclear
Confusion about how to respond / response mode
Letter was too vague / Was not sure what the letter was about
Others

Don't Know / Did Not Read It All / No Response

142

66
88%
3%
3%
2%
5%
0%

ACS Online Visual Testing

68.

Mail Instruction Card (Patriotic)

(INSTRUCTIONS) Please click on the image below in areas that caught your attention when
you first looked at this letter. This could be pictures, words, logos, etc. Please click three times.

(%)
Logo
Header
Paragraph 1
Required
Option 1
Option 2
Print Code
Multi-language
Paragraph 2

Click
1st Click

2nd Click

3rd Click

57%
0%
8%
26%
9%
1%
0%
0%
0%

24%
0%
1%
54%
18%
4%
0%
0%
0%

14%
2%
20%
11%
30%
21%
0%
0%
2%

C2_ Patriotic _Front

1st Click

143

ACS Online Visual Testing

2nd Click

3rd Click

144

ACS Online Visual Testing

69.

If you had to explain the purpose of the item you just saw to a friend, how would
you describe it to them? (CODED OPEN-ENDS)

Mail Instruction Card (Community)
Count (n-size)
More than one way to fill out the survey
Letter from the Census Bureau (mentioned Census specifically)
Mandatory / Required by law
Survey / Reminder to fill out a survey (did not mention Census)
Not legitimate / A scam / Not part of the regular census
Explaining the importance of filling out the survey/why they are conducting the survey
Online survey / Important to complete online
Junk mail / Spam / Garbage / Nonsense
Government snooping / Using threats / Intimidation
All other negatives
Others
Don't Know / No Response

70.

72
33%
25%
10%
8%
4%
3%
3%
1%
1%
1%
6%
4%

Thinking about the item you just read, were there any parts you found to be
unclear or confusing? (CODED OPEN-ENDS)

Mail Instruction Card (Community)
Count (n-size)
It was fine / Nothing was unclear
Why is this required by law / What happens if I do not respond
Everything was unclear
Confusion about how to respond / response mode
Why is Spanish-language materials included
Others

Don't Know / Did Not Read It All / No Response

145

57
88%
4%
2%
2%
2%
2%
2%

ACS Online Visual Testing

71.

Reminder Postcard (Patriotic Front)

(INSTRUCTIONS) Please click on the image below in areas that caught your attention when
you first looked at this letter. This could be pictures, words, logos, etc. Please click three times.

(%)
Logo
Stamp
Return 1
Paragraph 1
Link
Address
Code

72.

Click
1st Click

2nd Click

3rd Click

64%
0%
2%
1%
33%
1%
0%

23%
1%
2%
1%
63%
10%
0%

18%
2%
8%
5%
37%
30%
1%

1st Click

2nd Click

3rd Click

23%
35%
0%
39%
1%
3%
0%
0%

13%
11%
1%
46%
4%
20%
1%
4%

20%
11%
4%
15%
4%
32%
11%
4%

Reminder Postcard (Patriotic Back)
(%)
Paragraph 1
Image (Lincoln)
Paragraph 2
Required
Paragraph 3
Link
Paragraph 4
Footer

Click

(See next pages for heat maps)

146

ACS Online Visual Testing
B1_ Patriotic _Front

1st Click

2nd Click

3rd Click

147

ACS Online Visual Testing
B1_ Patriotic _Back

1st Click

2nd Click

3rd Click

148

ACS Online Visual Testing

73.

If you had to explain the purpose of the item you just saw to a friend, how would
you describe it to them? (CODED OPEN-ENDS)

Reminder Postcard (Patriotic)
Count (n-size)
Letter from the Census Bureau (mentioned Census specifically)
Mandatory / Required by law
More than one way to fill out the survey
Survey / Reminder to fill out a survey (did not mention Census)
Online survey / Important to complete online
Explaining the importance of filling out the survey/why they are conducting the survey
Survey helps determine the needs of my community / How funds are allocated
Letter about previous or soon to come letter / Instructions coming soon
Junk mail / Spam / Garbage / Nonsense
Government snooping / Using threats / Intimidation
Others
Don't Know / No Response

74.

124
25%
16%
15%
12%
8%
4%
4%
2%
2%
2%
5%
5%

Thinking about the item you just read, were there any parts you found to be
unclear or confusing? (CODED OPEN-ENDS)

Reminder Postcard (Patriotic)
Count (n-size)
It was fine / Nothing was unclear
Cluttered / Too many fonts/sizes/colors / Writing too small to read easily
Everything was unclear
Is it legitimate / Looks like junk mail
Why is this required by law / What happens if I do not respond
Confusion about how to respond / response mode
Others

Don't Know / Did Not Read It All / No Response

149

99
90%
3%
2%
2%
1%
1%
1%
0%

ACS Online Visual Testing

75.

Internet Instruction Card (Control)

(INSTRUCTIONS) Please click on the image below in areas that caught your attention when
you first looked at this letter. This could be pictures, words, logos, etc. Please click three times.

(%)
Logo
Header
Link
Address
Multi-language
Footer

Click
st

1 Click

2nd Click

3rd Click

64%
21%
5%
7%
0%
3%

27%
34%
16%
11%
2%
11%

13%
15%
18%
20%
2%
32%

A2_Control_Front

1st Click

2nd Click

150

ACS Online Visual Testing

3rd Click

76.

If you had to explain the purpose of the item you just saw to a friend, how would
you describe it to them? (CODED OPEN-ENDS)

Internet Instruction Card (Control)
Count (n-size)
Letter from the Census Bureau (mentioned Census specifically)
Online survey / Important to complete online
Survey / Reminder to fill out a survey (did not mention Census)
Explaining the importance of filling out the survey/why they are conducting the survey
Survey helps determine the needs of my community / How funds are allocated
Mandatory / Required by law
Junk mail / Spam / Garbage / Nonsense
Unnecessary / Useless
Others
Don't Know / No Response

77.

63
29%
22%
21%
6%
3%
2%
2%
2%
5%
10%

Thinking about the item you just read, were there any parts you found to be
unclear or confusing? (CODED OPEN-ENDS)

Internet Instruction Card (Control)
Count (n-size)
It was fine / Nothing was unclear
Is it legitimate / Looks like junk mail
Letter was too vague / Was not sure what the letter was about
Who is this organization / Never heard of ACS
Why is Spanish-language materials included
Everything was unclear
How was I 'Randomly selected'
Why is this required by law / What happens if I do not respond
Others

Don't Know / Did Not Read It All / No Response
151

54
72%
7%
4%
4%
4%
2%
2%
2%
2%
2%

ACS Online Visual Testing

78.

Mail Instruction Card (Control)

(INSTRUCTIONS) Please click on the image below in areas that caught your attention when
you first looked at this letter. This could be pictures, words, logos, etc. Please click three times.

(%)
Logo
Header
Paragraph 1
Required
Option 1
Paragraph 2
Multi-language
Option 2
Footer

Click
1st Click

2nd Click

3rd Click

52%
12%
8%
26%
0%
1%
0%
0%
0%

10%
24%
7%
38%
15%
6%
0%
0%
0%

9%
2%
22%
29%
22%
15%
0%
0%
0%

C2_ Control _Front

1st Click

152

ACS Online Visual Testing

2nd Click

3rd Click

153

ACS Online Visual Testing

79.

If you had to explain the purpose of the item you just saw to a friend, how would
you describe it to them? (CODED OPEN-ENDS)

Mail Instruction Card (Control)
Count (n-size)
More than one way to fill out the survey
Survey / Reminder to fill out a survey (did not mention Census)
Letter from the Census Bureau (mentioned Census specifically)
Online survey / Important to complete online
Letter about previous or soon to come letter / Instructions coming soon
Not legitimate / A scam / Not part of the regular census
Junk mail / Spam / Garbage / Nonsense
Government snooping / Using threats / Intimidation
Others
Don't Know / No Response

80.

58
43%
24%
14%
7%
5%
2%
2%
2%
2%
0%

Thinking about the item you just read, were there any parts you found to be
unclear or confusing? (CODED OPEN-ENDS)

Mail Instruction Card (Control)
Count (n-size)
It was fine / Nothing was unclear
Everything was unclear
Letter was too vague / Was not sure what the letter was about
Confusion about how to respond / response mode
Who is this organization / Never heard of ACS
Others

Don't Know / Did Not Read It All / No Response

154

60
93%
2%
2%
2%
2%
0%
0%

ACS Online Visual Testing

81.

Reminder Postcard (Control Front)

(INSTRUCTIONS) Please click on the image below in areas that caught your attention when
you first looked at this letter. This could be pictures, words, logos, etc. Please click three times.

(%)
Return 1
Stamp
Return 2
Print Code
Address

82.

Click
1st Click

2nd Click

3rd Click

82%
4%
7%
1%
5%

29%
16%
32%
0%
23%

22%
13%
30%
4%
31%

1st Click

2nd Click

3rd Click

0%
24%
66%
2%
7%
1%
0%
0%

1%
21%
33%
7%
31%
4%
2%
1%

3%
3%
13%
15%
36%
14%
14%
2%

Reminder Postcard (Control Back)
(%)
Print Code
Logo
Return 1
Greeting
Paragraph 1
Paragraph 2
Paragraph 3
Signature

Click

(See next pages for heat maps)

155

ACS Online Visual Testing
B1_ Control _Front

1st Click

2nd Click

3rd Click

156

ACS Online Visual Testing
B1_ Control _Back

1st Click

2nd Click

3rd Click

157

ACS Online Visual Testing

83.

If you had to explain the purpose of the item you just saw to a friend, how would
you describe it to them? (CODED OPEN-ENDS)

Reminder Postcard (Control)
Count (n-size)
Letter from the Census Bureau (mentioned Census specifically)
Survey / Reminder to fill out a survey (did not mention Census)
Survey helps determine the needs of my community / How funds are allocated
More than one way to fill out the survey
Explaining the importance of filling out the survey/why they are conducting the survey
Letter about previous or soon to come letter / Instructions coming soon
Online survey / Important to complete online
Mandatory / Required by law
Junk mail / Spam / Garbage / Nonsense
Unnecessary / Useless
Government snooping / Using threats / Intimidation
Randomly selected to do the survey
Not legitimate / A scam / Not part of the regular census
Others
Don't Know / No Response

84.

122
29%
18%
15%
7%
4%
3%
3%
2%
2%
2%
2%
1%
1%
7%
6%

Thinking about the item you just read, were there any parts you found to be
unclear or confusing? (CODED OPEN-ENDS)

Reminder Postcard (Control)
Count (n-size)
It was fine / Nothing was unclear
Addressed to 'resident' / Not addressed to me personally
Why is this required by law / What happens if I do not respond
When is it due / How long do we get to complete it
Who is this organization / Never heard of ACS
Is it legitimate / Looks like junk mail
Others

Don't Know / Did Not Read It All / No Response

158

97
94%
2%
1%
1%
1%
1%
0%
0%

ACS Online Visual Testing

Attributes
(ALL SAW THE FOLLOWING) Thank you. The mail exercises are complete. You will
now see a few brief questions based on what you have just seen.
Thinking about all the mail items you viewed, please indicate how much you believe the
following words apply to the mail package you viewed today.
% “Applies completely”

(%)
Official
Important
Easy-to-understand
Attention-grabbing
Urgent
Informative
Trustworthy
85.

Count (n-size)
Applies completely
Applies somewhat
Applies a little
Does not apply at all
Don't know

Community

Patriotic

Control

60
53
50
47
41
36
32

64
59
51
55
52
39
34

56
50
49
43
36
36
33

62
56
50
50
42
36
35

58
44
49
39
34
35
28

All

Package
Official

Community

Patriotic

Control

2009

509

500

501

499

60
25
9
3
2

64
20
10
2
3

56
29
9
3
2

62
24
8
3
2

58
28
8
3
2

Official

Community

Patriotic

Control

2009

509

500

501

499

53
29
12
4
2

59
23
12
4
2

50
32
11
4
2

56
26
12
4
2

44
34
13
6
2

Official

Community

Patriotic

Control

2009

509

500

501

499

50
34
11
3
1

51
32
11
3
1

49
34
11
3
2

50
35
10
3
1

49
34
11
3
1

Important
(%)
Count (n-size)
Applies completely
Applies somewhat
Applies a little
Does not apply at all
Don't know

87.

Official

Official
(%)

86.

ALL

All

Package

Easy-to-understand
(%)
Count (n-size)
Applies completely
Applies somewhat
Applies a little
Does not apply at all
Don't know

All

159

Package

ACS Online Visual Testing

88.

Attention-grabbing
(%)
Count (n-size)
Applies completely
Applies somewhat
Applies a little
Does not apply at all
Don't know

89.

Count (n-size)
Applies completely
Applies somewhat
Applies a little
Does not apply at all
Don't know

Community

Patriotic

Control

2009

509

500

501

499

47
32
14
5
1

55
26
13
4
1

43
35
15
5
1

50
33
11
4
1

39
35
18
6
1

Official

Community

Patriotic

Control

2009

509

500

501

499

41
32
16
7
2

52
29
12
5
1

36
34
19
8
2

42
31
18
6
2

34
34
17
10
3

Official

Community

Patriotic

Control

2009

509

500

501

499

36
36
18
7
1

39
32
18
7
2

36
39
17
5
2

36
36
18
7
1

35
37
18
8
1

Official

Community

Patriotic

Control

2009

509

500

501

499

32
34
18
8
5

34
31
20
8
6

33
33
19
8
5

35
37
15
8
4

28
36
20
10
5

All

Package

Informative
(%)
Count (n-size)
Applies completely
Applies somewhat
Applies a little
Does not apply at all
Don't know

91.

Package
Official

Urgent
(%)

90.

All

All

Package

Trustworthy
(%)
Count (n-size)
Applies completely
Applies somewhat
Applies a little
Does not apply at all
Don't know

All

160

Package

ACS Online Visual Testing

Closing Questions
92.

Overall, how would you describe your feelings about the American Community Survey?

(%)
Count (n-size)
Very favorable
Somewhat favorable
Somewhat unfavorable
Very unfavorable
Don't know

93.

Count (n-size)
Very likely
Somewhat likely
Somewhat unlikely
Very unlikely
Don't know

Official

Community

Patriotic

Control

2009

509

500

501

499

27
45
10
7
10

23
46
10
8
12

31
43
9
6
10

28
45
12
5
9

26
46
10
7
10

All

Package
Official

Community

Patriotic

Control

2009

509

500

501

499

56
26
5
7
5

51
29
6
7
6

61
20
5
8
5

61
24
4
6
5

49
30
6
8
4

Do you believe that answering the American Community Survey could
[ROTATED: personally benefit you | personally harm you] or neither benefit
nor harm you?
(%)
Count (n-size)
Personally benefit you
Personally harm you
Neither benefit nor harm
Both benefit and harm
Don't know

95.

Package

How likely would you be to participate in the American Community Survey if
contacted by the Census Bureau?
(%)

94.

All

All

Package
Official

Community

Patriotic

Control

2009

509

500

501

499

26
3
54
5
12

23
5
54
4
13

28
3
49
6
13

28
2
55
3
12

23
3
57
6
10

Do you believe that answering the American Community Survey could
[ROTATED: benefit your community | harm your community] or neither
benefit nor harm your community?
(%)
Count (n-size)
Benefit your community
Harm your community
Neither benefit nor harm
Both benefit and harm
Don't know

All

Package
Official

Community

Patriotic

Control

2009

509

500

501

499

46
2
31
7
13

41
2
34
7
15

50
2
29
7
11

47
2
31
5
14

46
2
31
8
11

161

ACS Online Visual Testing

Demographics (from Interviews)
(ALL SAW THE FOLLOWING) Thank you. The following final questions are for statistical
purposes only.

96.

As best as you can recall, did you or someone else in your household complete
the 2010 Census?
(%)
Count (n-size)
I completed the 2010 Census
Someone else in my household
completed the 2010 Census
No one in my household completed
the 2010 Census
Don't know

97.

Package
Official

Community

Patriotic

Control

2009

509

500

501

499

73

70

77

73

71

7

8

7

7

7

7

6

6

7

7

13

16

11

12

14

As best as you can recall, have you or someone else in your household
completed the American Community Survey?
(%)
Count (n-size)
I have completed the American
Community Survey
Someone else in my household has
completed the American Community
Survey
No one in my household has
completed the American Community
Survey
Don't know

98.

All

All

Package
Official

Community

Patriotic

Control

2009

509

500

501

499

13

14

13

13

11

2

3

1

3

1

49

45

46

49

56

36

37

39

34

32

Which of the following do you use to access the Internet?
(%)
Count (n-size)
Laptop computer
Desktop computer
Cell phone
Tablet
Other

All

Package
Official

Community

Patriotic

Control

2009

509

500

501

499

65
62
42
30
1

67
61
39
26
1

64
63
43
32
1

64
62
44
30
1

64
63
44
31
1

162

ACS Online Visual Testing

99.

Which of the following is your primary way of accessing the Internet?
(%)
Count (n-size)
Desktop computer
Laptop computer
Cell phone
Tablet
Other

100.

Count (n-size)
None
1-2 hours per day
3-4 hours
5-6 hours
7-8 hours
9-10 hours
More than 10 hours
Don't know

Community

Patriotic

Control

2009

509

500

501

499

46
43
6
4
0

46
44
6
3
0

46
44
5
5
0

47
41
7
4
1

47
42
6
4
0

All

Package
Official

Community

Patriotic

Control

2009

509

500

501

499

1
40
27
13
7
5
4
1

2
39
30
14
7
3
3
2

1
41
29
12
6
5
4
2

2
43
24
14
7
4
5
1

2
38
26
13
9
6
5
1

Were you born in the United States?
(%)
Count (n-size)
Yes
No
Prefer not to answer

102.

Package
Official

On an average day, about how many hours do you use the internet at home,
work or somewhere else?
(%)

101.

All

All

Package
Official

Community

Patriotic

Control

2009

509

500

501

499

91
8
1

91
7
1

90
8
2

91
8
0

90
8
1

Would you describe the area where you live as urban, suburban, or rural?
(%)
Count (n-size)
Urban
Suburban
Rural
Prefer not to answer

All

Package
Official

Community

Patriotic

Control

2009

509

500

501

499

24
52
22
2

22
53
22
2

26
51
20
2

26
51
20
2

21
52
24
2

163

ACS Online Visual Testing

Demographics (Piped in from Panel Data)
103.

HISPANIC ETHNICITY (CODED FROM SAMPLE)
(%)
Count (n-size)
No, I am not
Yes, Mexican, Mexican-American,
Chicano
Yes, Puerto Rican
Yes, Cuban, Cuban American
Yes, Central or South American
Yes, Other Spanish/Hispanic/Latino

104.

Count (n-size)
White
Black or African American
Asian
2+ races
American Indian or Alaska Native
Native Hawaiian/Pacific Islander

Community

Patriotic

Control

2009

509

500

501

499

89

89

90

89

89

5

5

6

6

5

1
1
2
2

2
1
2
1

1
1
1
1

1
0
2
1

1
1
1
3

All

Package
Official

Community

Patriotic

Control

2009

509

500

501

499

80
12
4
3
0
0

79
12
5
4
0
0

81
11
4
3
0
1

80
11
4
4
0
0

80
12
4
2
0
0

Official

Community

Patriotic

Control

2009

509

500

501

499

46
54

46
54

45
55

46
54

46
54

GENDER (CODED FROM SAMPLE)
(%)
Count (n-size)
Male
Female

106.

Package
Official

RACE (CODED FROM SAMPLE)
(%)

105.

All

All

Package

AGE, 7-WAY (BASED ON BIRTH YEAR, CODED FROM SAMPLE)
(%)
Count (n-size)
17 and under
18-24
25-34
35-44
45-54
55-64
65 and over

All

Package
Official

Community

Patriotic

Control

2009

509

500

501

499

0
1
19
19
16
22
23

0
0
17
21
15
24
23

0
0
20
18
17
22
23

0
0
22
18
18
20
22

0
1
19
19
17
21
23

164

ACS Online Visual Testing

107.

AGE, 4-WAY (BASED ON BIRTH YEAR, CODED FROM SAMPLE)
(%)
Count (n-size)
18-29
30-44
45-64
65+

108.

Count (n-size)
Less than high school
High school
Some college
Bachelor's degree or higher

Count (n-size)
Yes
No

Patriotic

Control

2009

509

500

501

499

13
26
38
23

13
26
38
23

13
26
39
23

13
26
38
22

13
26
38
23

All

Package
Official

Community

Patriotic

Control

2009

509

500

501

499

3
23
29
44

4
25
27
44

4
23
28
45

4
25
28
43

2
20
31
46

All

Package
Official

Community

Patriotic

Control

2009

509

500

501

499

88
12

89
11

87
13

89
11

89
11

HOUSEHOLD SIZE (CODED FROM SAMPLE)
(%)
Count (n-size)
1
2
3
4
5
6
7+

111.

Community

CONSIDERS SELF HEAD OF HOUSEHOLD (CODED FROM SAMPLE)
(%)

110.

Package
Official

HIGHEST EDUCATION LEVEL (CODED FROM SAMPLE)
(%)

109.

All

All

Package
Official

Community

Patriotic

Control

2009

509

500

501

499

22
41
16
13
4
2
1

24
38
17
14
5
1
0

21
44
16
12
4
2
1

20
44
16
14
4
1
1

24
38
16
14
4
4
0

MARITAL STATUS (CODED FROM SAMPLE)
(%)
Count (n-size)
Married
Never married
Divorced
Living with partner
Widowed
Separated

All

Package
Official

Community

Patriotic

Control

2009

509

500

501

499

58
16
11
7
6
2

59
15
12
7
5
2

59
13
13
7
6
2

60
16
9
8
6
1

56
19
10
7
6
2

165

ACS Online Visual Testing

112.

113.

HOUSEHOLD: ADULT MEMBERS, 18+ YEARS OLD (CODED FROM SAMPLE)
(%)

All

Count (n-size)
Working - as a paid employee
Not working - retired
Working - self-employed
Not working - disabled
Not working - looking for work
Not working - on temporary layoff from
a job
Not working - other

Community

Patriotic

Control

2009

509

500

501

499

53
23
7
5
4

51
23
10
5
4

53
22
7
4
4

55
21
6
6
4

55
24
6
4
5

1

1

0

1

1

7

6

9

7

6

HOUSEHOLD INCOME (CODED FROM SAMPLE)
(%)
Count (n-size)
Less than $5,000
$5,000 to $7,499
$7,500 to $9,999
$10,000 to $12,499
$12,500 to $14,999
$15,000 to $19,999
$20,000 to $24,999
$25,000 to $29,999
$30,000 to $34,999
$35,000 to $39,999
$40,000 to $49,999
$50,000 to $59,999
$60,000 to $74,999
$75,000 to $84,999
$85,000 to $99,999
$100,000 to $124,999
$125,000 to $149,999
$150,000 to $174,999
$175,000 or more

114.

Package
Official

All

Package
Official

Community

Patriotic

Control

2009

509

500

501

499

1
1
1
2
2
3
4
5
5
6
7
10
11
7
7
13
7
4
6

2
0
1
3
2
3
5
5
4
6
7
9
11
7
6
13
7
4
6

1
1
1
1
2
3
5
5
6
6
6
9
9
7
10
13
6
4
5

1
1
2
2
2
4
2
6
4
6
6
11
9
6
5
12
8
4
8

1
1
0
2
2
3
3
4
3
5
10
10
14
6
9
12
6
4
5

REGION 4 - BASED ON STATE OF RESIDENCE (CODED FROM SAMPLE)
(%)
Count (n-size)
South
West
Midwest
Northeast

All

Package
Official

Community

Patriotic

Control

2009

509

500

501

499

35
25
23
17

35
23
24
18

35
26
21
17

35
24
24
17

33
27
23
17

166

ACS Online Visual Testing

115.

OWNERSHIP STATUS OF LIVING QUARTERS (CODED FROM SAMPLE)
(%)
Count (n-size)
Owned or being bought by you or
someone in your household
Rented for cash
Occupied without payment of cash
rent

116.

Count (n-size)
Yes
No

Count (n-size)
None
1
2

Patriotic

Control

2009

509

500

501

499

73

73

73

72

75

25

25

24

26

23

2

2

3

2

2

All

Package
Official

Community

Patriotic

Control

2009

509

500

501

499

92
8

90
10

93
7

89
11

94
6

All

Package
Official

Community

Patriotic

Control

2009

509

500

501

499

97
3
0

96
4
0

98
2
0

95
5
0

97
2
0

HOUSEHOLD: CHILDREN, 2-5 YEARS OLD (CODED FROM SAMPLE)
(%)
Count (n-size)
None
1
2
3
4

119.

Community

HOUSEHOLD: CHILDREN, 0-1 YEAR OLD (CODED FROM SAMPLE)
(%)

118.

Package
Official

HOUSEHOLD PREVIOUSLY HAD INTERNET (CODED FROM SAMPLE)
(%)

117.

All

All

Package
Official

Community

Patriotic

Control

2009

509

500

501

499

92
7
2
0
0

92
6
2
0
0

92
6
2
0
0

92
7
1
0
0

92
6
2
0
0

HOUSEHOLD: CHILDREN, 6-12 YEARS OLD (CODED FROM SAMPLE)
(%)
Count (n-size)
None
1
2
3
4

All

Package
Official

Community

Patriotic

Control

2009

509

500

501

499

87
8
4
1
0

86
9
4
0
0

88
7
3
1
0

87
8
4
0
0

86
9
5
1
0

167

ACS Online Visual Testing

120.

HOUSEHOLD: CHILDREN, 13-17 YEARS OLD (CODED FROM SAMPLE)
(%)
Count (n-size)
None
1
2
3

121.

All

Package
Official

Community

Patriotic

Control

2009

509

500

501

499

89
9
2
0

87
10
2
0

91
7
2
1

90
8
2
0

88
10
2
0

HOUSEHOLD: ADULT MEMBERS, 18+ YEARS OLD (CODED FROM SAMPLE)
(%)
Count (n-size)
1
2
3
4
5
6
7

All

Package
Official

Community

Patriotic

Control

2009

509

500

501

499

24
57
13
4
1
0
0

27
57
12
3
1
0
0

23
58
15
3
1
0
0

21
62
11
4
2
0
0

27
52
13
6
2
0
0

(ALL SAW THE FOLLOWING) Thank you, that concludes the survey. If you have any
questions or comments about this survey you may send to Sam Hagedorn at
[email protected].

168