ACS MP IMPS and ROIL NSC- ROIL Study Plan Attachment D

American Community Survey Research and Evaluation Program
December 10, 2020

ACS Research & Evaluation Analysis Plan (REAP)

Regional Office
Internet Letter Test

THIS PAGE INTENTIONALLY BLANK

i

U.S. Census Bureau

TABLE OF CONTENTS
1.

INTRODUCTION ................................................................................................................ 1

2.

BACKGROUND .................................................................................................................. 1
2.1
2.2

3.

ACS Data Collection Strategy ....................................................................................... 1
Letters available to Field Representatives ................................................................... 2

LITERATURE REVIEW ......................................................................................................... 4
3.1

Mail Materials ............................................................................................................. 4
3.1.1
3.1.2
3.1.3

3.2
4.

CAPI response rates..................................................................................................... 5

RESEARCH QUESTIONS AND METHODOLOGY ................................................................... 7
4.1
4.2

Sample Design ............................................................................................................. 7
Experimental Design.................................................................................................... 7
4.2.1
4.2.2

4.3
4.4

Address-Side Message................................................................................... 8
Response Option ........................................................................................... 8

Research Questions ..................................................................................................... 9
Analysis Metrics........................................................................................................... 9
4.4.1
4.4.2
4.4.3
4.4.4
4.4.5
4.4.6
4.4.7

5.

Required by Law Messaging ............................ Error! Bookmark not defined.
Due Date Messaging...................................................................................... 4
TQA as a response option.............................................................................. 5

Self-Response Response Rate for the mailable CAPI Universe ...................... 9
CAPI Response Rate for the mailable CAPI Universe ..................................... 9
Overall Response Rate for the mailable CAPI Universe ............................... 10
CAPI Outcome Codes................................................................................... 10
Average Number of Contact Attempts ........................................................ 10
Standard Error of the Estimates .................................................................. 11
Exploratory Analysis .................................................................................... 11

ASSUMPTIONS AND LIMITATIONS .................................................................................. 11
5.1
5.2

Assumptions .............................................................................................................. 11
Limitations ................................................................................................................. 12

6.

TABLE SHELLS.................................................................................................................. 12

7.

POTENTIAL CHANGES TO ACS ......................................................................................... 13

8.

REFERENCES ................................................................................................................... 13

Appendix A. Regional Office Internet Letter – Atlanta Example ............................................. 14
Appendix B. Revised Regional Office Internet Letter with Past Due and only Internet
Option – Atlanta Example ............................................................................................... 16
Appendix C. Revised Regional Office Internet Letter with Past Due and TQA Response
Option – Atlanta Example ............................................................................................... 18
ii

U.S. Census Bureau

Appendix D. Revised Regional Office Internet Letter with Your Response is Required by
Law and only Internet Option – Atlanta Example ............................................................ 20
Appendix E. Revised Regional Office Internet Letter with Your Response is Required by
Law and TQA Response Option – Atlanta Example ......................................................... 22

iii

U.S. Census Bureau

1. INTRODUCTION
The U.S. Census Bureau continually evaluates how the American Community Survey (ACS) mail
contact materials and data collection methodology can be improved to increase survey
participation and reduce survey costs. Recent tests have shown that self-response increases
when mailings contain less text and are written in plain language. Increased self-response can
substantially decrease survey costs and improve data quality (Risley and Berkley, Forthcoming).
During the Computer-Assisted Personal Interviewing (CAPI) month, there are a series of letters
available to field representatives to provide to nonrespondent households to motivate them to
respond. One of these letters is the Internet Letter, which provides an internet user ID and
instructions on how to respond online. Until May 2020, the Internet Letter was not
implemented consistently across the regions. Some regions sent the Internet Letter to all
mailable CAPI cases and others sent it at the discretion of the field representative. However,
starting in May 2020 all the regional offices started sending the Internet Letter to all mailable
CAPI cases.
A new version of the Internet Letter was recently developed as part of a project to update the
design and messaging in the letters used by field representatives during CAPI. The update was
consistent with the design of the other ACS mail materials which are less text dense and use
plain language to increase the likelihood of survey cooperation. We were confident enough in
the changes made to the letter that the decision was made to implement it in production
without field testing.
The purpose of the Regional Office Internet Letter Test is to study variations on the new
Internet Letter to understand which content options are most effective at increasing selfresponse and combating decreasing CAPI response rates.

2. BACKGROUND
2.1

ACS Data Collection Strategy

The ACS contact strategy is detailed below to provide context for the field test.

1

U.S. Census Bureau

The initial sample consists of mailable, unmailable, and undeliverable addresses. The first two
mailings are sent to all mailable addresses in the monthly sample. The first mailing is a package
that includes a letter, a multilingual brochure, and a card with instructions on how to respond
via the internet. The letter contains an invitation to participate in the ACS online and more
information in a frequently asked questions format on the back of the letter. A week later, the
same addresses are sent a second mailing (reminder letter in a pressure seal mailer).
Responding addresses are removed from the address file after the second mailing to create a
new mailing universe of nonrespondents; these addresses are sent the third and fourth
mailings.1 The third mailing is a package that includes a letter, a paper questionnaire, and a
business reply envelope. Four days later, these addresses are sent a fourth mailing (reminder
postcard) which encourages them to respond.
After the fourth mailing, responding addresses are again removed from the address file to
create a new mailing universe of nonrespondents. The remaining addresses are sent the fifth
mailing (a urgent final reminder letter with a due date in a pressure seal mailer).
Two to three weeks later, responding addresses are removed and unmailable and undeliverable
addresses (from the initial sample) are added to create the universe of addresses eligible for
the CAPI nonresponse followup operation. Of this universe, a subsample is chosen to be
included in the CAPI operation. CAPI interviews start at the beginning of the month following
the fifth mailing. Field representatives attempt to interview those selected for CAPI by phone. If
they cannot reach them by phone, or do not have a phone number, they visit the addresses to
conduct in-person interviews.
In October 2020, we began mailing a letter, the Internet Letter, from the National Processing
Center (NPC) to all mailable addresses in the CAPI universe that provides information on how to
respond online in order to avoid an in-person visit. The letter arrives sometime during the first
week of CAPI data collection. In some cases, the letter will arrive prior to contact by a field
representative; but in others the letter will arrive after the initial contact. This letter is sent as a
pressure seal mailer.
Additional information about the ACS data collection strategy can be found in the ACS Design
and Methodology Report (U.S. Census Bureau 2014).
2.2

Letters available to Field Representatives

During the CAPI month, one method used by field representatives to address respondent
concerns and gain cooperation, is to send letters with additional information. These housing
unit CAPI letters can be requested by field representatives based on the situations they
1

Addresses deemed “undeliverable as addressed” (UAA) by the United States Postal Service are also removed
from the address files for subsequent mailings.

2

U.S. Census Bureau

encounter. For example, a field representative who cannot access a locked building may
request a “management letter” that is designed to address concerns raised by apartment
managers. In total, the field representatives have 12 letters and brochures available to them,
four of which they carry with them and can leave at the door. The remaining eight can be sent
to the respondent by the regional office at the request of the field representative.
One of the most often used letters is the Internet Letter, which encourages an online response
and provides internet login instructions to the recipient. An example of the Internet Letter that
has been used for several years is shown in Appendix A. The primary concern with the letter
was that it featured a large amount of text that is not connected to the primary purpose of the
letter and is unlikely to be salient with the reader. This can mean recipients are more likely to
overlook the instructions on responding online or the benefits of responding.
We decided to make changes to the Internet letter in a two-step process. First, we designed a
new letter as part of a project to update the CAPI letters sent from the regional offices (Work
Request #PI20-2-1417), shown in Appendix B, and implemented it in production in October
2020. We decided to implement the new letter without first doing a field test because based on
lessons learned from communications and survey literature as well as the results of recent ACS
testing of self-response materials we had strong evidence that it would be more effective than
the existing letter (Oliver, Heimel, and Schreiner, 2017; Risley and Berkley, Forthcoming).
The second step in the process is this field test, the Regional Office Internet Letter Test, which
tests two aspects of the letters to determine which content option will ultimately be most
effective at increasing self-response and combating decreasing CAPI response rates. Currently
the letter uses a “Past Due” message on the outside of the mailer and provides an internet
response option inside. The two changes being tested are:
•
•

The use of a “Required by Law” message instead of a “Past Due” message on the outside
of the mailer.
Providing TQA as a response option in addition to the internet response option.

In previous testing, a stated due date as well as a “Required by Law” message has been found
to increase self-response (Risley and Berkley, Forthcoming; Risley and Oliver, Forthcoming). By
comparing the two we are trying to determine which is more effective for this mailing
Providing TQA as a response option for the ACS has been considered for some time. The
primary benefit of providing it as a response option during CAPI is that TQA responses are on
average cheaper than CAPI responses. However, they are more expensive than self-responses
by internet and mail and there is a concern that enough of an increase in calls could overwhelm
the call centers.

3

U.S. Census Bureau

3. LITERATURE REVIEW
3.1

Mail Materials

3.1.1

Mail Material Design

In the 2018 Mail Materials Test, we tested a series of design changes and variations on the use
of mandatory messaging (Risley and Berkley, Forthcoming). The goals of the letter design
changes were:
•
•
•

Emphasize the Census Bureau brand in ACS mail materials.
Use visual design principles to draw attention to key messages.
Create a consistent look and feel across all mailings.

To meet these goals, the following changes were made to the self-response mailings:
•
•
•

The Census Bureau logo was moved to the top left corner of letters and envelope to
increase prominence.
Where possible, bulleted lists were used in order to decrease the amount of text in the
letters.
Bolding and call-out boxes were used in order to better emphasize key parts of the
letter.

The treatment that performed the best included these new design elements and features that
emphasized mandatory messaging. The redesigned Internet Letter makes use of these same
design principles.
3.1.2

Due Date Messaging

Census Bureau research suggests that the inclusion of a due date can boost survey selfresponse. The 2019 ACS Due Date Test was conducted to test a due date in the in the fifth
mailing (Risley and Oliver, Forthcoming). The study tested three locations for the due date:
•
•
•

In the call-out box on the outside of the envelope.
In the call-out box inside the letter.
In a contextual message that describes the consequences of not responding by the due
date.

The treatment that placed the due date both inside the letter and on the outside of the
envelope was found to be most effective in increasing self-response. Additionally, both the
“add” message and the “remove” message were found to be effective, with the “remove”
message being slightly more effective (Risley and Oliver, Forthcoming).
Due to its success, the due date was implemented into ACS production in November 2020. The
other message being tested in the Regional Office Internet Letter Test connects back to this due
4

U.S. Census Bureau

date by informing recipients that their response is now “Past Due”. By reminding recipients of
the Internet Letter that their response is late we attempt to connect back to the sense of
urgency created by the due date and create a connection to the consequence messaging.
3.1.3

TQA as a response option

Currently all mail materials provide the TQA number but do not state that it is a response
option, only that help is available by calling. By comparing the demographics of those that
currently respond by telephone and those that respond by personal interview, we see that,
based on 2018 ACS 1-year estimates, there are overlaps of key characteristics that suggest
members of the CAPI universe would respond by telephone if they knew it was an option. Some
of the key characteristics include income, internet access, and primarily speaking a language
other than English (U.S. Census Bureau 2020). 2
Table 1. Demographics of Responders
Demographic
Percent living below
poverty line
Percent of households
reporting no internet access
Percent that do not speak
English “very well”

All Modes

Telephone
Interview

Personal
Interview

12.9%

17.1%

19.8%

12.0%

41.8%

19.1%

8.4%

13.5%

13.0%

Source: U.S. Census Bureau, “People and Households Represented in Each American Community Survey Data Collection Mode”
data visualization, https://www.census.gov/library/visualizations/interactive/acs-collection.html

3.2

CAPI response rates

The CAPI response rates have been tracked since the inception of the full ACS in 2005. Shown in
Figure 1, for about the first ten years of ACS CAPI data collection, response rates were in the
mid-90s. However, over time the response rate in CAPI began to drop. To combat this, field
representatives were given the option of sending letters to addresses (see Background section).
Unfortunately, the response rate in CAPI has continued to decline and currently fluctuates
around 80 percent. As shown in Figure 2, during the same time period, internet response during
the CAPI month has been increasing. In 2013, the portion of internet responses received during
the CAPI month was less than 1 percent. In the last year, it accounts for around 4.5 to 6 percent
of internet responses.

2

Starting in October 2020, the new Internet letter includes Spanish text at the bottom of the letter with
instructions on how to respond to the survey in Spanish by phone or online. The is a new feature, although we
are not testing the Spanish language with this experiment.

5

U.S. Census Bureau

Figure 1: CAPI Response Rates from 2005-2020

Source: U.S. Census Bureau, American Community Survey

Figure 2: Portion of Internet Response Rate Received in Last Month of Data Collection

Source: U.S. Census Bureau, American Community Survey

6

U.S. Census Bureau

Note: The spike at the beginning of 2019 is due to the ending of the government shutdown.

4. RESEARCH QUESTIONS AND METHODOLOGY
4.1

Sample Design

The Regional Office Internet Letter test will be conducted using the June 2021 ACS production
sample, with mailout at the end of July. The monthly ACS production sample of approximately
295,000 addresses is divided into 24 methods panel groups, where each group contains
approximately 12,000 addresses. Each methods panel group is a representative subsample of
the entire monthly sample and each monthly sample is representative of the entire yearly
sample and the country.
For this test, the universe of analysis will be limited to the mailable portion of the CAPI
universe. This portion of each methods panel group contains approximately 2,600 addresses.
Each of the four treatments will each be assigned to six methods panel groups. Hence, each
treatment will have a sample size of approximately 15,600 addresses. The control treatment
will be the only treatment sent the production materials.
4.2

Experimental Design

This experimental design isolates each of the two content factors being studied. One factor is
the message on the outside of the pressure seal mailer: a “Past Due” message or a “Required by
Law” message (Address-Side Message). The other content factor is the inclusion of TQA as a
response option (Response Option). This experiment uses a fully factorial design with one
control treatment and three experimental treatments.
Table 2: Regional Office Internet Test Experimental Treatments
Treatment

Address-Side Message

Response Option

Control

Past Due

Internet Only

Experimental
Treatment 1

Past Due

Internet + TQA

Experimental
Treatment 2

Required by Law

Internet Only

Experimental
Treatment 3

Required by Law

Internet + TQA

7

U.S. Census Bureau

4.2.1

Address-Side Message

Starting in the October 2020 ACS panel there is a response due date provided in the fifth
mailing. As part of this test, we are testing if the message on the outside of the letter should
reference back to this due date or use the “Your Response is Required by Law” message, which
is used on many of the other mail materials. To determine which message is more effective we
are testing two variations on the messaging uses on the address-side:
•

The message that their response is past due:

•

The message that their response is required by law:

4.2.2

Response Option

Currently all mail materials mention the TQA number but do not state that it is a response
option. We assume that there are respondents whose first choose would be to respond using
the telephone, however it is more expensive than if they were to respond online or by mail and
there is concern that the call volume would be too much for the current TQA operation to
handle. However, since a TQA response would still be cheaper than a CAPI response and the
internet letter is being sent to a small universe, TQA as a response option is being tested as part
of this test.
•

Internet is the only response option provided:

•

Internet and TQA are both provided as response options:

8

U.S. Census Bureau

4.3

Research Questions
1. What is the effect of using Past Due message vs a Required by Law message on the
address side of the pressure seal mailer?
2. What is the effect of using an internet response option only vs an internet and TQA
response option inside the letter?

4.4
4.4.1

Analysis Metrics
Self-Response Response Rate for the mailable CAPI Universe

Calculating the self-response rates, for the addresses that were mailed the Internet letter,
allows us to evaluate the effectiveness of the treatments in increasing self-response during the
CAPI month.

Self-Response
Response Rate

=

Number of mailable and deliverable sample addresses in the CAPI
universe that either provided a nonblank3 return by mail, a return
by TQA, or a complete or sufficient partial4 response by internet
Total number of mailable, deliverable, and eligible5 sample
addresses in the CAPI universe6

* 100

Additionally, the individual TQA rate will be analyzed over time and the TQA call volume will be
monitored.
4.4.2
CAPI Response Rate for the mailable CAPI Universe
Calculating the CAPI response rates, for the addresses that were mailed the Internet letter,
allows to better understand differences in the self-response. In cases where the letter design
convinces more respondents to response online rather than wait for an interview, with no
other effect, there should be a corresponding decrease in CAPI response.

9

U.S. Census Bureau

CAPI Response
Rate

Number of mailable and deliverable sample addresses in the CAPI
universe that provided a CAPI interview
Total number of mailable, deliverable, and eligible3 sample
addresses in the CAPI universe4

=

* 100

4.4.3
Overall Response Rate for the mailable CAPI Universe
The overall response rate for the addresses that were mailed the Internet Letter, provides the
final piece that allows a complete look at the letter design’s effect on response.

Overall
Response
Rate

4.4.4

Number of mailable and deliverable sample addresses in
the CAPI universe that either provided a nonblank5 return by mail,
a return by TQA, a complete or sufficient partial6 response by
internet, or a CAPI interview
Total number of mailable, deliverable, and eligible7 sample
addresses in the CAPI universe8

=

* 100

CAPI Outcome Codes

Calculating the rate at which CAPI cases ended in specific CAPI outcome codes, including
refusals and late mail and internet returns, allows us to evaluate the effectiveness of the
treatments.

CAPI Outcome
Code Rate

4.4.5

=

Number of mailable and deliverable sample addresses in the
CAPI universe that have a specific final CAPI outcome code
Total number of mailable, deliverable, and eligible9 sample
addresses in the CAPI universe10

* 100

Average Number of Contact Attempts

The average number of interviewer contact attempts during the CAPI operation will be
compared as an additional measure to understand a treatment’s effect on the CAPI operation.
This will include both personal visits as well as phone contacts. The average will be across the
3

Business addresses, addresses under construction, etc. are not eligible.
We remove addresses deemed to be Undeliverable-as-Addressed by the Postal Service if no response is
received.
5 A blank form is a form in which there are no persons with sufficient response data and there is no telephone
number listed on the form.
6
A sufficient partial internet response is one in which the respondent reached the Pick Next Person screen for a
household with two or more individuals on the roster or has gone through the place of birth question for a
1-person household.
7
Business addresses, addresses under construction, etc. are not eligible.
8
We remove addresses deemed to be Undeliverable-as-Addressed by the Postal Service if no response is
received.
9
Business addresses, addresses under construction, etc. are not eligible.
10
We remove addresses deemed to be Undeliverable-as-Addressed by the Postal Service if no response is
received.
4

10

U.S. Census Bureau

entire mailable CAPI universe, with those self-responding before the first interviewer contact
counting as zero contacts.
4.4.6

Standard Error of the Estimates

We will estimate the variances of the point estimates and differences using the Successive
Differences Replication (SDR) method with replicate weights – the standard method used in the
ACS (see U.S. Census Bureau, 2014, Chapter 12). In calculating the different rates, we will use
replicate subsampling adjusted weights, which account for the initial sampling probabilities and
the subsampling during the CAPI operation. We will calculate the variance for each rate and for
the difference between rates using the formula below:
4
Var (X0 ) =
80

80

(Xr - X0 )2
r=1

Where:
Xr = the estimate calculated using the rth replicate
X0 = the estimate calculated using the full sample
The standard error of the estimate (X0) is the square root of the variance.
4.4.7

Exploratory Analysis

There will be additional exploratory analysis performed. Areas that will be part of the
exploratory analysis include differences based on Regional Office as well as differences based
on if the first interviewer contact was before or after the Internet Letter was received.
Additionally, the use of logistic regression to isolate treatment effects will be explored.

5. ASSUMPTIONS AND LIMITATIONS
5.1

Assumptions
•
•
•

A single ACS monthly sample is representative of an entire year (twelve panels) and the
entire frame sample, with respect to both response rates and cost, as designed.
A single methods panel group (1/24 of the full monthly sample) is representative of the
full monthly sample, as designed.
We assume that there is no difference between treatments in mail delivery timing or
subsequent response time. The treatments had the same sample size and used the
same postal sort and mailout procedures. Previous research indicated that postal
procedures alone could cause a difference in response rates at a given point in time
between experimental treatments of different sizes, with response for the smaller

11

U.S. Census Bureau

treatments lagging (Heimel 2016).
5.2

Limitations
•
•

Group quarters and sample housing unit addresses from remote Alaska and Puerto Rico
are not included in the sample for the test.
The relative cost analysis uses estimates to make cost projections. These estimates do
not account for monthly variability in production costs, such as changes in staffing,
production rates, or printing price adjustments.

6. TABLE SHELLS
This table shell will be used to compare the self-response of any two treatments.
Table 3. Sample Table for Comparing Two Self-Response Response Rates
Treatment A
Treatment B
Difference
Overall Self-Response
####
####
####

P-Value
####

Internet

####

####

####

####

Mail

####

####

####

####

TQA

####

####

####

####

Source: U.S. Census Bureau, American Community Survey, Regional Office Internet Field Letter Test
Note: Minor additive discrepancies are due to rounding. Standard errors are in parentheses. An asterisk (*) indicates a
statistically significant result. Significance was tested based on a two tailed t-test at the α=0.1 level

This table shell will be used to compare any two treatments on any single metric such as the
refusal rate or the average number of contact attempts.
Table 4. Sample Table for Comparing Two Rates
Metric
Treatment A – Treatment B
Treatment A
####
--Treatment B
####
####

P-Value
--####

Source: U.S. Census Bureau, American Community Survey, Regional Office Internet Field Letter Test
Note: Minor additive discrepancies are due to rounding. Standard errors are in parentheses. An asterisk (*) indicates a
statistically significant result. Significance was tested based on a two tailed t-test at the α=0.1 level.

This table shell will be used to compare the overall response rate for any two treatments.

12

U.S. Census Bureau

Table 5. Sample Table for Comparing Overall Response Rates by mode
Treatment A
Treatment B
Difference
Overall Response
####
####
####
Self-Response
####
####
####
Internet
####
####
####
Mail
####
####
####
TQA
####
####
####
CAPI
####
####
####

P-Value
####
####
####
####
####
####

Source: U.S. Census Bureau, American Community Survey, Regional Office Internet Field Letter Test
Note: Minor additive discrepancies are due to rounding. Standard errors are in parentheses. An asterisk (*) indicates a
statistically significant result. Significance was tested based on a two tailed t-test at the α=0.1 level.

7. POTENTIAL CHANGES TO ACS
This research could potentially change the letter design of the newly implemented Internet
Letter.

8. REFERENCES
Heimel, S. (2016). “Postal Tracking Research on the May 2015 ACS Panel,” 2016 American
Community Survey Research and Evaluation Report Memorandum Series #ACS16-RER01, April 1, 2016.
Oliver, B., Heimel, S., and Schreiner, J. (2017). “Strategic Framework for Messaging in the
American Community Survey Mail Materials.” Washington, DC: U.S. Census Bureau.
Retrieved on May 6, 2020 from
https://www.census.gov/content/dam/Census/library/workingpapers/2017/acs/2017_Oliver_01.pdf
Risley, M. & Berkley, J. (forthcoming). “2018 Mail Materials Test.” Washington DC, U.S. Census
Bureau.
Risley, M. & Oliver, B. (forthcoming). “2019 ACS Due Dates Test.” Washington DC, U.S. Census
Bureau.
U.S. Census Bureau (2014). “American Community Survey Design and Methodology, Chapter
12: Variance Estimation.” Retrieved on September 6, 2019 from
https://www.census.gov/programs-surveys/acs/methodology/design-andmethodology.html
U.S. Census Bureau (2020). “People and Households Represented in Each American Community
Survey Data Collection Mode.” Retrieved on November 20, 2020 from
https://www.census.gov/library/visualizations/interactive/acs-collection.html

13

U.S. Census Bureau

Appendix A. Previous Production Regional Office Internet Letter – Los Angeles
Example

14

U.S. Census Bureau

15

U.S. Census Bureau

Appendix B. New Production Regional Office Internet Letter with Past Due and
only Internet Option – Los Angeles Example

16

U.S. Census Bureau

17

U.S. Census Bureau

Appendix C. Revised Regional Office Internet Letter with Past Due and TQA
Response Option – Los Angeles Example

18

U.S. Census Bureau

19

U.S. Census Bureau

Appendix D. Revised Regional Office Internet Letter with Your Response is
Required by Law and only Internet Option – Los Angeles Example

20

U.S. Census Bureau

21

U.S. Census Bureau

Appendix E. Revised Regional Office Internet Letter with Your Response is
Required by Law and TQA Response Option – Los Angeles Example

22

U.S. Census Bureau

23

U.S. Census Bureau