2020 Census Experiment: Optimization of Self-Response in the 2020 Census Experiment Study Plan

The memorandum and attached document(s) was prepared for Census Bureau internal use. If
you have any questions regarding the use or dissemination of the information, please contact
the Stakeholder Relations Staff at [email protected].

2020 CENSUS PROGRAM INTERNAL MEMORANDUM SERIES: 2019.24.i(rev)
Date:

June 3, 2019

MEMORANDUM FOR: The Record
From:

Deborah M. Stempowski (signed June 3, 2019)
Chief, Decennial Census Management Division

Subject:

2020 Census Experiment: Optimization of Self-Response in the 2020 Census
Experiment Study Plan

Contact:

Jennifer Reichert
Decennial Census Management Division
301-763-4298
[email protected]

This memorandum releases the final version of the 2020 Census Experiment: Optimization of SelfResponse in the 2020 Census Experiment Study Plan, which is part of the 2020 Census Program for
Evaluations and Experiments (CPEX). This updated study plan replaces the version originally released
on June 3, 2019.For specific content related questions, you may also contact the authors:
Kelly Mathews
Decennial Statistical Studies Division
301-763-5639
[email protected]

Michael Bentley
Decennial Statistical Studies division
301-763-4306
[email protected]

Gina Walejko
Decennial Statistical Studies Division
301-763-1643
[email protected]

Sarah Konya
Decennial Statistical Studies Division
301-763-9835
[email protected]

Julia Coombs
Decennial Statistical Studies Division
301-763-5747
[email protected]

Ioana (Julia) Marasteanu
Decennial Statistical Studies Division
301-763-8827
[email protected]

census.gov

2020 Census Experiment
Optimization of Self-Response in the 2020
Census Experiment
Study Plan

Kelly Mathews, Decennial Statistical Studies Division
Sarah Konya, Decennial Statistical Studies Division
Michael Bentley, Decennial Statistical Studies Division
Julia Coombs, Decennial Statistical Studies Division
Gina Walejko, Decennial Statistical Studies Division
Ioana (Julia) Marasteanu, Decennial Statistical Studies
Division
Steven Scheid, Decennial Statistical Studies Division

August 7, 2019
Version 1.5

Page intentionally left blank.

Optimizing Self-Response in the 2020 Census Experiment, Version 1.5

Table of Contents
I.

Introduction .............................................................................................................................. 1

II.

Background ............................................................................................................................... 1

III.

Assumptions ............................................................................................................................. 7

IV.

Research Questions .................................................................................................................. 7

V.

Methodology............................................................................................................................. 8

VI.

Data Requirements ................................................................................................................. 22

VII.

Risks ........................................................................................................................................ 22

VIII.

Limitations .............................................................................................................................. 22

IX.

Issues That Need to be Resolved........................................................................................... 23

X.

Division Responsibilities ....................................................................................................... 23

XI.

Milestone Schedule ................................................................................................................ 23

XII.

Review/Approval Table ......................................................................................................... 24

XIII.

Document Revision and Version Control History ............................................................... 24

XIV. Glossary of Acronyms ........................................................................................................... 24
XV.

References............................................................................................................................... 25

XVI. Appendix A............................................................................................................................. 27

Optimizing Self-Response in the 2020 Census Experiment, Version 1.5

List of Tables
Table 1. 2020 Census Self-Response Mailing Strategy with In-Home Dates .................................. 7
Table 2. Descriptions of the Panels ................................................................................................... 10
Table 3. General Sequence of Mailings for the Panels .................................................................... 10
Table 4. Approximate In-Home Delivery Mailing Dates for the Panels ........................................ 11
Table 5. Panels Eligible for Assignment for each Stratum .............................................................. 12
Table 6. Self-Response Rates for the Control Panel and 2010 Census Materials and 2010 Census
Mailing Timing Panel by Mode ................................................................................................ 13
Table 7. Response Rates over Time .................................................................................................. 13
Table 8. Self-Response Rates for the Control Panel and 2020 Census Materials without Internet
Option Panel by Mode ............................................................................................................... 14
Table 9. Self-Response Rates for the Control Panel, Internet Choice Cases Given Internet First
Contact Strategy Panel, Internet First Cases Given Internet Choice Contact Strategy Panel
by Mode ...................................................................................................................................... 15
Table 10. Self-Response Rates for the 2010 Census Materials and 2010 Census Mailing Timing
Panel, and 2020 Census Materials without Internet Option Panel by Mode .......................... 16
Table 11. Self-Response Rates for the Two Control Panels ............................................................ 17
Table 12. Self-Response Rates for Control Panel and Internet First Housing Units Not Receiving
Mailed Materials Panel .............................................................................................................. 18
Table 13. Self-Response Rates for Control Panel, Internet Choice Cases Given Internet First
Contact Strategy Panel, and Housing Units in Internet Choice Tracts Not Receiving Mailed
Materials Panel ........................................................................................................................... 18
Table 14. Self-Response Rates for Control Panel and Housing Units in Internet First Tracts Not
Receiving Mailed Materials Panel ............................................................................................ 19
Table 15. Self-Response Rates for Control Panel, Internet Choice Cases Given Internet First
Contact Strategy, and Housing Units in Internet First Tracts Not Receiving Mailed
Materials Panel ........................................................................................................................... 20

Optimizing Self-Response in the 2020 Census Experiment, Version 1.5

I.

Introduction

Self-response in the 2020 Census includes several major differences relative to the 2010 Census,
including: 1. An internet response option that allows for response with or without a unique
Census ID; 2. Tailored mail contact strategies in which different geographic areas receive
different mail materials for a total of up to five mailings; 3. A telephone number for Census
Questionnaire Assistance (CQA) available on all mailings that allow callers to provide their
census data by phone; 4. Responses in multiple modes, using multiple device types, and in
multiple languages; 5. An integrated partnership and communications program that features local
and national partnerships and micro targeted advertising to promote self-response; 6. Instead of
delivering the mailings to the whole country on a single day, each of the five mailings will be
stretched out over a one-week period into groups of four different mail cohorts. All of these
response features are a part of the innovation area called Optimizing Self-Response (OSR) for
the 2020 Census, and they primarily apply to the Type of Enumeration Area (TEA) called SelfResponse. Other TEAs will be contacted and enumerated, but are out of the scope of this
experiment.
This experiment seeks to understand the net impacts of the OSR innovation area in the 2020
Census. In particular, the focus is to evaluate the quantitative impacts of the mailing strategy, the
overall influence of the internet response option, and the impact of the communications
campaign on internet and phone response. Therefore, the following experimental treatments will
be tested (strategies discussed in further detail in the next section):





II.

A sample of housing units will receive the 2010 Census mail strategy.
A sample of housing units will receive the 2020 Census mail strategy with no mention of
the option to respond via the internet.
A sample of housing units will have their mail contact strategy switched between Internet
Choice and Internet First.
A sample of housing units will receive no direct mailings from the Census Bureau.

Background

Maximizing rates of self-response, both overall (paper, internet, and Census Questionnaire
Assistance) and on the internet, is an important way to lower cost for the 2020 Decennial Census.
Any household that does not self-respond will be sent to Nonresponse Followup, which is a
much costlier operation. Among the self-response modes, internet self-response is the most costeffective. Results of prior OSR research embedded within earlier census tests or other survey
operations, contributed to the design of the contact strategies for the 2020 Census. Middecade
testing between 2010 and 2020 was designed to improve self-response rates, with the 2010
Census as the starting point. Discussion of the results from the 2010 Census and the middecade
tests follows.
2010 Census. The 2010 Census used a mailing strategy consisting of multiple contacts, which
included the use of a replacement questionnaire, a second questionnaire sent to housing units that
1

Optimizing Self-Response in the 2020 Census Experiment, Version 1.5

had already received the first questionnaire, not used in the 2000 Census. The multiple contacts
in the 2010 Census included:






An advance letter alerting households that the census questionnaire would be sent to them
soon.
The initial questionnaire.
A direct mail postcard to select areas, providing information to Chinese, Korean,
Vietnamese, and Russian speaking households on how to obtain assistance in completing
their questionnaire.
The reminder postcard, which served as a reminder to those who had not yet mailed back
their questionnaire, and as a thank you to those who had.
An English-only replacement questionnaire distributed in one of three treatments:
o Blanket replacement mailing – where all households within the treatment area
received a replacement questionnaire, regardless of whether or not they sent back
their initial questionnaire.

.
o Target replacement mailing – where households within treated areas only received
a replacement questionnaire if their initial questionnaire had not been received
and scanned into the system by a predetermined date. Postal tracking data were
used to identify Undeliverable as Addressed that were not yet checked in, so that
the addresses were not sent a questionnaire.
o No replacement mailing – where households within treated areas did not receive a
replacement questionnaire, regardless of whether or not they sent back their initial
questionnaire.
The final mail response rate after the Nonresponse Followup operation was approximately 67
percent, with approximately 3 percentage points coming from the replacement questionnaire.
Mail response rates followed a similar pattern to mail response rates from the 2000 Census,
which did not have a replacement mailing. Results also indicated that a full-scale replacement
questionnaire ought to be mailed to the entire country, not just certain areas. (Letourneau, 2012).
To determine which housing units would receive a bilingual Spanish/English questionnaire in the
initial mailout of the 2010 Census, data from the 2005-2007 American Community Survey
(ACS) were used. Spanish language assistance is defined as those housing units in which at least
one adult (age 15 or older) in the household speaks Spanish and does not speak English “very
well.” An entire geographic tract is identified as bilingual, meaning that all housing units in the
tract receive the bilingual mailing materials, if the number of Spanish language assistance
housing units is at least 20 percent of the occupied units in the tract. This threshold was
operationally feasible as well since the number of bilingual questionnaires produced was
budgeted to be approximately 10 percent of the mailout workload, or about 13 million housing
units. Data from the 2000 Census long form were used for comparison and analytical purposes
(Bentley, 2008). The same threshold was used in subsequent middecade tests.
Unlike the design of the 2020 Census, the 2010 Census: 1. Did not include an internet response
option; 2. Offered telephone interviews only during a limited window of time; 3. Had fewer
2

Optimizing Self-Response in the 2020 Census Experiment, Version 1.5

mailings to encourage response; 4. Provided a replacement paper questionnaire to some
geographic areas only; 5. Provided a Be Counted Form option for people who may not have
received a questionnaire or felt they were not counted; 6. Offered assistance in fewer languages;
7. Included an integrated partnership and communications program but with less emphasis on
partnerships and targeted advertising to promote responses; 8. Delivered the mailings to the
entire country in a single day. In this experiment, the 2010 Census Panel will be similar to that
of the actual 2010 Census but will exclude the Be Counted Form option and include the 2020
Census integrated partnership and communications program that emphasizes partnerships and
targets advertising to promote responses.
2012 National Census Test. In the 2012 National Census Test, six contact strategy panels were
experimentally tested, in addition to other content testing. The Internet Push panel, which did not
include the advance letter used in the 2010 Census and included a second reminder postcard
before the questionnaire, had a positive impact on both internet and overall self-response rates.
As a result, this panel was recommended for self-response in future testing (Bentley et al., 2014).
Because of timing and resource constraints, the 2012 National Census Test was only able to test
materials and the internet response mode in English. It was therefore recommended to test the
inclusion of another language in order to determine whether response patterns differ for those
with limited English proficiency (Bentley et al., 2014). This was done in the 2015 National
Content Test with the testing of three language panels using Spanish as the second language.
2013 American Community Survey. The American Community Survey (ACS) gave respondents
the opportunity to use the internet to self-respond for the first time in 2013. The ACS used a push
notification strategy to encourage internet self-response. In this strategy, respondents were first
presented with an internet invitation before receiving a paper questionnaire. Analysis of the first
quarter ACS 2013 data investigated whether hard-to-interview groups (as defined by Joshipura,
2008) were more likely to self-respond when presented an internet option. The analysis showed
that households with a respondent over 65 years old and households with a respondent without a
high school education had statistically significant lower self-response proportions (by 3.32 and
8.06 percentage points, respectively) when given the internet option, and not given a paper
questionnaire until approximately two weeks later (Nichols, Horwitz, and Tancreto, 2014). In an
attempt to improve the self-response rates for these specific groups, three OSR panels in the
2015 National Content Test provided the mail questionnaire earlier.
2014 Census Test. The 2014 Census Test used the successful Internet Push strategy from the
2012 National Census Test as the control panel, since this panel had the highest overall weighted
response rate (64.8 percent) of the six panels tested. Other Internet Push panels for the 2014
Census Test were modified versions of this panel. The modifications included replacing postal
mail invitations, as well as postal reminders, with email reminders.
Three of the eight panels tested in the 2014 Census Test were sent the initial invitation and first
reminder via email in lieu of postal mail. None of these three panels had a higher response rate
than the panels that did not receive the invitation and first reminder via email (Bentley and

3

Optimizing Self-Response in the 2020 Census Experiment, Version 1.5

Rothhaas, 2015). This suggests that email is not an effective replacement for postal mail as an
invitation strategy.
Other studies have researched the impact of sending invitations and reminders via email on
response. One recent study conducted a longitudinal survey in the United Kingdom and found
that while the invitations and reminders sent via email did not affect response propensity, email
contact did increase the likelihood that those who responded did so on the internet (Cernat and
Lynn, 2014).
2015 Optimizing Self-Response Test. One important aspect of this test was that it was the first
time the Census Bureau looked at targeting housing units for what would eventually become
Internet First and Internet Choice. This began as a method, tested in the Savannah, GA
Designated Marketing Area, that aimed to enhance sampling stratification using internet
connection data from the Federal Communications Commission (FCC) and the low response
propensity score (LRS) from the Planning Database (PDB). The study tested whether using the
internet connection data provided by the FCC, or the LRS data provided by the PDB, was better
or the same at predicting overall and internet response rates (Mathews and Rothhaas, 2015). The
use of the FCC and LRS data helped us to make the contact strategy assignments in later tests.
This test also delivered a robust communications campaign that included paid TV, radio, print,
out-of-home, and digital advertisements. This allowed the Census Bureau to understand the
source of internet responses (i.e. from mail materials versus traditional advertisements versus
digital ads).
2015 National Content Test. The 2015 National Content Test tested modifications to the timing,
order, type and number of contacts for the Internet Push strategy. It also investigated whether
sending the first reminder via email and postal mail (rather than just by email) had an impact on
self-response rates, particularly internet self-response rates. Building on previous research, the
2015 National Content Test stratified the sample and tailored contact strategies based on
information about the prevalence of internet connectivity from the FCC and the estimated LRS
from the PDB (Mathews, 2015).
Results from the OSR portion of the 2015 National Content Test show that a fifth mailing should
be implemented, as sending a third postcard reminder resulted in the highest overall response
rates for the panels. Another finding is that housing units in low response areas which received a
paper questionnaire in the first mailing, had a significantly higher overall response rate than all
but one of the other panels, suggesting that using the Internet Choice contact strategy in low
response areas increases overall self-response. One panel of the 2015 National Content Test did
not receive a paper questionnaire, which resulted in the lowest overall response rate. This result
suggests that not providing a paper questionnaire negatively affects overall self-response rates.
This test also found that the first reminder postcard should be sent earlier to improve response
among Spanish-speaking households, and that the swim-lane or dual-sided letter format should
be used (Phelan, 2016).
2016 Census Test. The 2016 Census Test expanded on middecade research by: 1. Testing the
replacement of the first reminder postcard with a letter; 2. Sending a brochure instead of a letter;
4

Optimizing Self-Response in the 2020 Census Experiment, Version 1.5

and 3. Including a language insert with the letter. The analysis of the 2016 Census Test found
that self-response rates improve when using a letter instead of a postcard in the second mailing,
and when including a language insert with the first mailing, especially for limited English
proficiency households (Coombs, Lestina, and Phelan, 2017).
2017 Census Test. The 2017 Census Test was the first middecade test to incorporate the updated
definition of the Internet Choice contact strategy. The tracts that received the Internet Choice
contact strategy for the 2017 Census Test were those that fell into the following definition
(Mathews and Phelan, 2018):
Any tract that, based on American Community Survey self-response rates, responds by mail
more than by internet and has at least one of the following additional attributes:




Is a low responding tract when an Internet First strategy is used, based on American
Community Survey self-response rates1 (<41.3 percent overall self-response rate).
Is a tract with higher older population, based on American Community Survey estimates
(at least 22 percent of the population is age 65 and over).
Is a tract with less internet access, based on FCC data (no more than 400 household
internet connections, per 1,000 households in the tract).

The remaining tracts received the Internet First contact strategy (previously referred to as
Internet Push). Results for the 2017 Census Test are forthcoming.
2018 End-to-End Census Test. The cut-off values used for the 2017 Census Test Internet Choice
definition were reevaluated and it was determined that the FCC cutoff should be adjusted. This
adjustment would allow for a more accurate representation of the availability of internet access
for tracts that prefer responding by mail rather than by internet.

1

Any tracts missing American Community Survey self-response rates were automatically assigned to the Internet
Choice contact strategy.
5

Optimizing Self-Response in the 2020 Census Experiment, Version 1.5

The tracts chosen to receive the Internet Choice contact strategy for the 2018 End-to-End Census
Test were those that fell into the following definition (Mathews and Phelan, 2018):
Any tract that, based on ACS self-response rates, responds by mail more than by internet and has
at least one of the following additional attributes:




Is a low responding tract when an Internet First strategy is used, based on American
Community Survey self-response rates1 (<41.3 percent overall self-response rate).
Is a tract with higher older population, based on ACS estimates (at least 22 percent of the
population is age 65 and over).
Is a tract with less internet access, based on FCC data (no more than 600 household
internet connections, per 1,000 households in the tract).

The remaining tracts received the Internet First contact strategy. The forthcoming 2018 OSR
Operational Assessment will contain some results of the 2018 End-to-End Census Test.
All of the middecade research helped construct the contact strategies for the 2020 Census (see
Table 1). The Internet First contact strategy is divided into four cohorts that will all receive the
same mailing materials but at different times in order to reduce call volumes for the Census
Questionnaire Assistance centers. Additionally, if a tract contains more than 20 percent of
housing units that have Spanish language assistance needs, according to 5-year ACS estimates,
the tract will receive mailing materials that are bilingual. A household that has Spanish language
assistance needs is defined as a household that has at least one person (age 15 or older) who
speaks Spanish and does not speak English “very well.” Otherwise, the mailing materials will be
in English with a short message in Spanish about how to respond.
Note that for this experiment, all mailings (treatment and control) will be mailed at once rather
than in cohorts.

6

Optimizing Self-Response in the 2020 Census Experiment, Version 1.5

Table 1. 2020 Census Self-Response Mailing Strategy with In-Home Dates
Mailing
2

Mailing 1

Mailing 3*

Mailing 4*

Mailing 5*
“It’s not too
late”
Postcard

Letter (Internet
First) or
Letter +
Questionnaire
(Internet Choice)

Letter

Postcard

Letter +
Questionnaire

Internet
First

1
2
3
4

3/12/2020
3/13/2020
3/19/2020
3/20/2020

3/16/2020
3/17/2020
3/23/2020
3/24/2020

3/26/2020
3/27/2020
4/02/2020
4/03/2020

4/08/2020
4/09/2020
4/15/2020
4/16/2020

4/20/2020
4/20/2020
4/27/2020
4/27/2020

Internet
Choice

N/A

3/13/2020

3/17/2020

3/27/2020

4/09/2020

4/20/2020

Panel

Cohort

(*) Targeted only to nonresponding households

III.

Assumptions

1. The Census Data Lake will contain 2020 Census responses and operational data required for
analysis.
2. Mail materials can be printed, assembled, and mailed by staff at the National Processing
Center.
3. The sample to assess the communications campaign can be removed from the mailing
stream.

IV.

Research Questions

1. What was the overall impact of all of the 2020 mailing strategy innovations, on self-response
rates and costs, compared to the 2010 Census strategy?
2. Does mentioning the option to respond on the internet yield higher self-response rates?
3. Were the Internet Choice and Internet First strategies efficiently targeted, and effective, in
improving response rates for those respective areas?
4. What is the impact of the 2020 mailing strategy, specifically (i.e., controlling for the addition
of the internet option), on self-response rates and costs, compared to the 2010 Census
strategy?

7

Optimizing Self-Response in the 2020 Census Experiment, Version 1.5

5. What is the impact of sending the bilingual mailing to all households in the experiment and
control groups (as opposed to English-only tracts receiving the English-only mailing, and
bilingual tracts receiving the bilingual mailing)?
6. What is the impact of the communications campaign on internet response in the absence of
any direct mailings from the Census Bureau? Among different audience segments of the
2020 campaign?
7. What is the impact of the communications campaign on phone response in the absence of any
direct mailings from the Census Bureau? Among different audience segments of the 2020
campaign?

V.

Methodology
A. Experimental Design

This experiment seeks to understand the net impacts of the OSR innovation area in the 2020
Census. In particular, the focus is on evaluating the quantitative impacts of the mailing strategy,
and on the overall influence of the internet response option. Specifically:
a. A sample of housing units will receive the 2010 Census mail strategy and timing.
This will provide a treatment to quantitatively compare all innovations in the 2020
mailing strategy to the 2010 approach, including: additional mailings, the addition of an
internet option, the phone number provided in all mailings, and the timing of receiving
the materials. The 2010 materials will be updated, as appropriate, with necessary 2020
elements such as the logo, mandatory legal language, dates, and the 2020 survey
questions. Housing units in this sample will not be told about the internet instrument or
provided a URL to respond. However, if they hear about the internet option through the
communications campaign, word of mouth, or other general awareness, they could still
respond online (either without an ID, or with their ID that would still be present above
the barcode on the mail materials).
b. A sample of housing units will receive the 2020 Census mail strategy with no mention of
the option to respond via the internet.
This will provide a treatment to quantitatively compare the addition of the internet
response option to the 2020 Census. Housing units in this sample will receive all of the
2020 Census mailing materials, but will not mention the internet instrument or the URL
for response. However, if they hear about the internet option through the communications
campaign, word of mouth, or other general awareness, they could still respond online
(either without an ID, or with their ID that would still be present above the barcode on the
mail materials). These housing units will receive the 2020 Census mailing timing.

8

Optimizing Self-Response in the 2020 Census Experiment, Version 1.5

c. Switch the mail contact strategy for a sample of housing units between Internet Choice
and Internet First.
The Internet Choice strategy will be targeted to areas expected to benefit from a paper
questionnaire in the initial mailing. Based on extensive research, this includes census
tracts with relatively low overall self-response rates in the ACS that tend to respond by
paper more than by internet, tracts with an older population based on ACS estimates, and
tracts with lower internet subscribership based on data from the FCC. In order to evaluate
the effectiveness of this design, a sample of housing units in the Internet Choice areas
will be randomly selected to receive the Internet First strategy instead (and vice versa).
We will also control for the presence of Internet using data from ACS. If the opposite
mail strategy elicits higher response rates, or if the different strategies yield similar rates,
that may indicate that we need to reassess this design in the future. These housing units
will receive the 2020 Census mailing materials and timing.
d. A sample of housing units will be removed from the production mail stream of all
materials.
This will provide a treatment to measure the effect of the communications campaign (and
larger overall census environment) on internet and phone response. The campaign will
consist of many different components including paid advertising, earned media, Statistics
in Schools, and a local and national partnership program. Many of these communications
will include the URL to the census web form. Others will contain the CQA phone
numbers. This sample of Master Address File Identifiers (MAFIDs) will not receive any
direct mailings from the Census Bureau. Thus, any internet or phone responses received
will have come from one of the communications campaign components, word-of-mouth,
or general awareness. All treatment responses will be non-ID, but we hypothesize the
matching rates will be high given the sample comes from MAF extracts. All sampled
MAFIDs will be included in NRFU in the case that an internet or phone response is not
completed.
In order to answer the research questions, five experimental panels are planned. As a way to
minimize the burden on the printer, all households within the treatment groups will receive
bilingual materials. This blanketed language assignment for the treatment groups does not match
the production language assignment, in which ACS language data is used to determine if a tract
should receive English-only materials or bilingual materials. As a result, a defined control group
is needed for appropriate comparison. The control panel and the five experimental panels will
help in the evaluation of the OSR operation. Table 2 below has the description of all panels,
including the control. The experimental panels (E1 through E4) are compared to the control
panel (C2). A second panel (C1) is compared to C2 in order to determine the impact of using the
bilingual form for all tracts.

9

Optimizing Self-Response in the 2020 Census Experiment, Version 1.5

Table 2. Descriptions of the Panels
Panels
Description
C1: 2020 Census
All mail materials and mailing timing are the same as the materials
Materials
sent to production 2020 Census housing units.
C2: 2020 Census
All mail materials and mailing timing are the same as the materials
Materials (all
sent to production 2020 Census housing units. All households will
bilingual)
receive the bilingual form.
E1: 2010 Census
Mail materials and mailing timing will closely resemble the materials
Materials and 2010
sent to production 2010 Census housing units, with updates where
Census Mailing
needed. In particular, no mention of the internet instrument will be
Timing (all bilingual)
made in the materials.
All mail materials will not mention the internet instrument. The
E2: 2020 Census
mailing timing will be the same as the materials sent to production
Materials without
2020 Census housing units. Note that these housing units will receive
Internet Response
the Internet Choice contact strategy mailing regardless of their
Option (all bilingual)
originally assigned contact strategy.
E3: Internet Choice
Census tract was assigned to the Internet Choice strategy but will
Cases Given Internet
receive Internet First materials instead. The mailing timing will be
First Contact Strategy the same as the materials sent to production 2020 Census housing
(all bilingual)
units.
E4: Internet First
Census tract was assigned to the Internet First strategy but will
Cases Given Internet
receive Internet Choice materials instead. The mailing timing will be
Choice Contact
the same as the materials sent to production 2020 Census housing
Strategy (all bilingual) units.
E5: No mailings
All mail materials will be suppressed.
Table 3 and Table 4. illustrate the difference between the mail materials and mail timing for
Panels C1, C2, E1, E2, and E5. Table 3 shows the general sequence of mailings, while Table 4.
shows the in-home mailing delivery dates. Housing units in Panel E1 will receive the same type
of materials along with a similarly spaced mail timing as the 2010 Census. For this experiment,
the replacement questionnaire in the 2010 Census Mailing 4 will be sent to all nonresponding
housing units in Panel E1.
Table 3. General Sequence of Mailings for the Panels
2010 Census/
Panel E1
2020 Census/
C1, C2, E2,
E3, E4
E5

Mailing 1
Advance letter

Mailing 2
Initial
questionnaire
Reminder
letter

Letter w/internet
option
OR Questionnaire
w/internet option
na
na

(*) Targeted only to nonresponding households

10

Mailing 3
Reminder
postcard
*Reminder
postcard

Mailing 4
*Replacement
questionnaire
*Questionnaire
w/internet
option

Mailing 5

na

na

na

*“It’s not
too late”
postcard

Optimizing Self-Response in the 2020 Census Experiment, Version 1.5

Table 4. Approximate In-Home Delivery Mailing Dates for the Panels
Mailing 1
Mailing 2
Mailing 3
Mailing 4
Tuesday,
Tuesday,
Tuesday,
*Friday,
2010 Census/
March 10,
March 17,
March 24,
April 10,
Panel E1
2020
2020
2020
2020
Friday,
Tuesday,
*Friday,
*Thursday,
2020 Census/
March 17,
March 27,
April 9,
C1, C2, E2, E3, March 13,
2020
2020
2020
2020
E4
E5

N/A

N/A

N/A

N/A

Mailing 5

*Monday,
April 20,
2020
N/A

(*) Targeted only to nonresponding households

The OSR sample for this experiment will be selected at the same time as the National Mailing
sample for a similar experiment. The OSR sample and the National Mailing sample together
make up the Overall sample. The Overall sample selection along with details on the OSR sample
selection are presented below. For further details on the National Mailing sample, please see the
Extending the Census Environment to the Mail Materials Study Plan.
The Overall sample is preselected from the Self-Response TEA (TEA = 1) housing units.
Housing units with TEA = 1 from the MAF file are split up into seven groups, each of which is
representative of the United States. Tracts are fully contained within one of the seven groups.
This split into seven groups is necessary, as different experiments are conducted at different
geographic levels, which leads to nonrandom gaps within the population, and therefore potential
bias in the frame. One of the seven groups is then chosen to be the eligible universe for this
experiment and the national mailing portion of the Extending the Census Environment to the
Mail Materials experiment. Before selecting the Overall sample, the eligible universe is split into
two strata: Internet First and Internet Choice. Each stratum will be sorted by state, county,
language flag, tract, ineligibility flag, internet flag, and MAFID.
A random sample of 72,786 housing units will be selected from each of the two strata. Then,
within each stratum, one of the nine panels (five panels are for the OSR sample, E1-E5, two
panels are for the Mailing sample, M10 and M11, with the two control panels, C1 and C2, being
shared by both the OSR and Mailing sample) will be systematically assigned to a housing unit,
depending on the strata. See Table 5 for eligible panel assignments.
Other direct mailers outside of the five production mailings are in the process of being
developed. These mailers will be sent before or during self-response in order to promote
response in subpopulations of interest. Though the addresses identified to receive these
additional mailers will not be known until after sampling for this experiment is underway, it is
possible that the sample selected in this experiment may overlap with the additional mailers
being proposed. One of the proposed mailers may target up to 20 percent of all mailable
addresses. To account for this possible overlap in samples, all of the panels will be sampled at
120 percent of the minimum sample size calculated in the appendix.

11

Optimizing Self-Response in the 2020 Census Experiment, Version 1.5

Panels

Table 5. Panels Eligible for Assignment for each Stratum
Number Assigned from
Each Stratum
Internet
Internet
First
Choice
12,478
12,478
C1
12,478
12,478
C2
6,239
6,239
E1
6,239
6,239
E2
0
12,478
E3
12,478
0
E4
12,478
6,239
E5
12,478
12,478
M10
12,478
12,478
M11
Using an alpha of 0.10, beta of 0.20, and a detectable difference of 3 percentage points in selfresponse rates, the sample size for the OSR sample (C1, C2, E1, E2, E3, E4, E5) is 118,541
housing units. The OSR sample size was calculated using the conservative estimate of the 2020
Census response rate of 57 percent and an estimated design effect of 1.75 based on the design
effect found for the response rate variable in the 2015 National Content Test. See Appendix A
for further sampling details.
B. Answering Research Questions
1. What was the overall impact of all of the innovations of the 2020 Census mailing strategy
(panel C2), on self-response rates and costs, compared to the 2010 Census strategy (panel
E1)?
We expect that the 2020 strategy will have a higher self-response rate since the changes made to
the materials and mailing strategies were recommendations of the middecade testing, whose goal
was to improve self-response. Although unprecedented, if the 2020 response rates are not higher
than those in 2010, we will conduct a comprehensive reassessment of the 2020 mailing
strategies. The self-response rates will be calculated and compared for the two experimental
panels that manipulate mailing materials (panel E1 and C2). The following formula will be used
to calculate self-response rates:
Self-response rate =

Number of Self-Responding Housing Units in Sample
* 100 percent
Number of Housing Units in Sample

To calculate the response rate by mode, the numerator will only include in-sample responses
from that mode, but the denominator will remain as the number of housing units in the sample.
Table 6 will be produced to display response by mode, and a t-test will be used to determine if
the 2020 Census design yielded a higher total self-response rate than the 2010 Census design.

12

Optimizing Self-Response in the 2020 Census Experiment, Version 1.5

Table 6. Self-Response Rates for the Control Panel and 2010 Census Materials and 2010
Census Mailing Timing Panel by Mode
Panel
Mail SelfInternet SelfCQA SelfTotal SelfResponse Rate
Response Rate
Response Rate
Response Rate*
C2: 2020 Census
Materials (all
bilingual)
E1: 2010 Census
Materials and
2010 Census
Mailing Timing
(all bilingual)
*t-test will be performed on this statistic.

Table 7. shows the response rates over time, which can also be used to determine self-response
patterns for each panel.
Table 7. Response Rates over Time
Response Response Response Response
Response
Rate
Rate
Rate
Rate
Final
Final
Rate
Panel
after
after
after
after
Response Return
before
First
Second
Third
Fourth
Rate
Rate
NRFU
Mailing
Mailing
Mailing
Mailing
C2: 2020 Census Materials
(all bilingual)
E1: 2010 Census Materials
and 2010 Census Mailing
Timing (all bilingual)
E2: 2020 Census Materials
without Internet Response
Option (all bilingual)
E3: Internet Choice Cases
Given Internet First Contact
Strategy (all bilingual)
E4: Internet First Cases
Given Internet Choice
Contact Strategy (all
bilingual)
E5: No Mailings

13

Optimizing Self-Response in the 2020 Census Experiment, Version 1.5

Cost will be determined using the following formula.
Cost per Panel per treated case
Total Cost of treatment + ∆Nonresponse X Nonresponse Followup Cost
=
Number of Cases Receiving Treatment
2. Does mentioning the option to respond on the internet yield higher self-response rates?
We expect that mentioning the internet response option will result in a higher self-response rate.
The following formula will be used to calculate self-response rates:
Self-response rate =

Number of Self-Responding Housing Units in Sample
* 100 percent
Number of Housing Units in Sample

To calculate the response rate by mode, the numerator will only include in-sample responses
from that mode, but the denominator will remain as the number of housing units in the sample.
Table 8 will be produced to display response by mode, and a t-test will be used to determine if
the 2020 Census materials with an internet option displayed (panel C2) yielded a higher total
self-response rate than the 2020 Census materials without the internet option (panel E2)
displayed (see Table 8. ).
Table 8. Self-Response Rates for the Control Panel and 2020 Census Materials without
Internet Option Panel by Mode
Panel
Mail SelfInternet SelfCQA SelfTotal SelfResponse Rate
Response Rate
Response Rate
Response Rate*
C2: 2020 Census
Materials (all
bilingual)
E2: 2020 Census
Materials
without Internet
Response Option
(all bilingual)
*t-test will be performed on this statistic.

The communications campaign will make it virtually impossible to create an environment
without any mention of the internet. That limitation will apply to all panels. In addition to the
metrics mentioned above, we will examine the characteristics of households and people that
respond via the internet when that is not a given option.

14

Optimizing Self-Response in the 2020 Census Experiment, Version 1.5

3. Were the Internet Choice and Internet First strategies efficiently targeted, and effective,
in improving response rates for those respective areas?
We expect that the two contact strategies, Internet Choice and Internet First, were targeted
effectively resulting in an improvement in self-response rates in each of their respective areas.
Self-response rates will be calculated and compared for Internet Choice and Internet First
strategies between the sample assigned to receive the alternative strategy (panels E3 and E4) and
the respective baseline (Internet Choice or Internet First units from panel C2). The following
formula will be used to calculate self-response rates:
Self-response rate =

Number of Self-Responding Housing Units in Sample
* 100 percent
Number of Housing Units in Sample

To calculate the response rate by mode, the numerator will only include in-sample responses
from that mode, but the denominator will remain as the number of housing units in the sample. A
t-test will be used to determine if the Internet Choice baseline sample (from C2) yielded a higher
total self-response rate than the Internet Choice switched to Internet First alternative sample. A ttest will also be used to determine if the Internet First baseline sample (from C2) yielded a higher
self-response rate than the Internet First switched to Internet Choice alternative sample (see
Table 9).
Table 9. Self-Response Rates for the Control Panel, Internet Choice Cases Given Internet
First Contact Strategy Panel, Internet First Cases Given Internet Choice Contact Strategy
Panel by Mode
Panel
Mail SelfInternet SelfCQA SelfTotal SelfResponse Rate
Response Rate
Response Rate
Response Rate*
C2: 2020 Census
Materials (all
bilingual)
E3: Internet
Choice Cases
Given Internet
First Contact
Strategy (all
bilingual)
E4: Internet First
Cases Given
Internet Choice
Contact Strategy
(all bilingual)
*t-test will be performed on this statistic.

In addition to the metrics above, we will examine the characteristics of households and people
that respond from the experimental and control panels.
15

Optimizing Self-Response in the 2020 Census Experiment, Version 1.5

4. What is the impact of the 2020 mailing strategy, specifically (i.e., controlling for the
addition of the internet option), on self-response rates and costs, compared to the 2010
Census strategy?
Again, because of the changes made to the materials and mail strategy from results of middecade
testing, we expect that the 2020 Census panel without mention of the internet (panel E2) will
have a higher self-response rate than the 2010 Census panel (panel E1). The following formula
will be used to calculate self-response rates:
Self-response rate =

Number of Self-Responding Housing Units in Sample
* 100 percent
Number of Housing Units in Sample

To calculate the response rate by mode, the numerator will only include in-sample responses
from that mode, but the denominator will remain as the number of housing units in the sample.
Table 10 will be produced to display response by mode, and a t-test will be used to determine if
the 2020 Census materials without mention of an internet option displayed yielded a higher total
self-response rate than the 2010 Census materials.
Table 10. Self-Response Rates for the 2010 Census Materials and 2010 Census Mailing
Timing Panel, and 2020 Census Materials without Internet Option Panel by Mode
Panel
Mail SelfInternet SelfCQA SelfTotal SelfResponse Rate
Response Rate
Response Rate
Response Rate*
E1: 2010 Census
Materials and
2010 Census
Mailing Timing
(all bilingual)
E2: 2020 Census
Materials
without Internet
Response Option
(all bilingual)
*t-test will be performed on this statistic.

Cost will be determined using the following formula.
Cost per Panel per treated case
Total Cost of treatment + ∆Nonresponse X Nonresponse Followup Cost
=
Number of Cases Receiving Treatment

16

Optimizing Self-Response in the 2020 Census Experiment, Version 1.5

5. Is there a difference in response rate between the control group in which everyone
receives the bilingual mailing and the group in which English-only tracts receive the
English-only mailing and bilingual tracts receive the bilingual mailing)?
We do not expect to see a difference in the response rate between the two groups. We will
reassess if our results suggest otherwise.
Self-response
rate =

Number of Self-Responding Housing Units in Sample
* 100 percent
Number of Housing Units in Sample

To calculate the response rate by mode, the numerator will only include in-sample responses
from that mode, but the denominator will remain as the number of housing units in the sample.
Table 11 will be produced to display response by mode, and a t-test will be used to determine if
the 2020 Census materials with all bilingual forms (C2) yields a different total self-response rate
than the 2020 Census materials with the appropriate language assignments (C1).
Table 11. Self-Response Rates for the Two Control Panels
Panel
Mail SelfInternet SelfCQA SelfResponse Rate
Response Rate
Response Rate

Total SelfResponse Rate*

C1: 2020 Census
Materials
C2: 2020 Census
Materials (all
bilingual)
*t-test will be performed on this statistic.

6. What is the impact of the communications campaign on Internet response in the absence of
any direct mailings from the Census Bureau? Among different audience segments of the 2020
campaign?
This comparison will allow us to estimate the proportion of housing units that self-respond via
the non-ID Internet instrument in the absence of any mailed materials. Any difference in selfresponse rates between the experimental and control group will allow us to estimate the
effectiveness of the mailed materials. It will also allow us to estimate the effectiveness of the
partnership and communications campaign on non-ID internet and phone response in absence of
any mailings.
Internet
self-response rate =

Number of Self-Responding Housing Units via the
Internet in Sample
Number of Housing Units in Sample

17

* 100 percent

Optimizing Self-Response in the 2020 Census Experiment, Version 1.5

Housing units sampled for the No Mailings group (E5) will be drawn from tracts assigned to
both contact strategies (i.e., Internet First and Internet Choice), so they will be compared to
different control groups. E5 housing units drawn from Internet First tracts will be compared to
the Internet First units in the control group receiving standard 2020 materials (Internet First units
from panel C1, see Table 12). The E5 housing units drawn from Internet Choice tracts will be
compared to both the experimental group given Internet First materials in place of Internet
Choice as well as the standard control (E3 and Internet Choice units from panel C1, see Table
13). This design allows us to compare Internet First and Internet Choice panels in E5 to Internet
First and Internet Choice units that received only an invitation to respond via the Internet until
Mailing 4 (C1 and E3, respectively). It will also allow us to compare Internet First and Internet
Choice panels in E5 to units that receive “business as usual” 2020 mailings (C1).
Table 12. Self-Response Rates for Control Panel and Internet First Housing Units Not
Receiving Mailed Materials Panel
Panel
Mail Self- Internet Self- CQA SelfTotal SelfResponse
Response
Response
Response
Rate
Rate*
Rate
Rate
C1: 2020 Census Materials
(Internet First units)
E5: No Mailings (Internet
First units)
*t-test will be performed on this statistic.

Table 13. Self-Response Rates for Control Panel, Internet Choice Cases Given Internet
First Contact Strategy Panel, and Housing Units in Internet Choice Tracts Not Receiving
Mailed Materials Panel
Panel
Mail SelfInternet SelfCQA SelfTotal SelfResponse Rate Response Rate* Response Rate
Response Rate
C1: 2020 Census
Materials
(Internet Choice
units)
E3: Internet
Choice Cases
Given Internet
First Contact
Strategy (all
bilingual)
E5: No Mailings
(Internet Choice
units)
*t-test will be performed on this statistic.
18

Optimizing Self-Response in the 2020 Census Experiment, Version 1.5

Results can be further analyzed by separately calculating the response rates of respondents in
different audience segments. (Segments were developed as part of the 2020 Communications
Campaign; U.S. Census Bureau, in preparation.).
7. What is the impact of the communications campaign on phone response in the absence of
any direct mailings from the Census Bureau? Among different audience segments of the
2020 campaign?
This comparison will provide an estimate of the rate at which housing units self-respond via
CQA in the absence of other avenues of response. Housing units who receive no mailed
materials will only have the option of responding via CQA or Internet without an ID.
CQA
self-response rate =

Number of Self-Responding Housing Units via the
CQA in Sample
Number of Housing Units in Sample

* 100 percent

The comparisons between treatment and control groups are the same as in the previous section.
Table 14. Self-Response Rates for Control Panel and Housing Units in Internet First Tracts
Not Receiving Mailed Materials Panel
Panel
Mail SelfInternet SelfCQA SelfTotal SelfResponse Rate
Response Rate
Response Rate* Response Rate
C1: 2020 Census
Materials
(Internet First
Tracts)
E5: No Mailings
(Internet First
Tracts)
*t-test will be performed on this statistic.

19

Optimizing Self-Response in the 2020 Census Experiment, Version 1.5

Table 15. Self-Response Rates for Control Panel, Internet Choice Cases Given Internet
First Contact Strategy, and Housing Units in Internet First Tracts Not Receiving Mailed
Materials Panel
Panel
Mail SelfInternet SelfCQA SelfTotal SelfResponse Rate
Response Rate
Response Rate* Response Rate
C1: 2020 Census
Materials
(Internet Choice
units)
E3: Internet
Choice Cases
Given Internet
First Contact
Strategy (all
bilingual)
E5: No Mailings
(Internet Choice
units)
*t-test will be performed on this statistic.

As with Question 7, this analysis will be repeated after grouping sampled housing units by
audience segment assignment. By doing so, we can test how audience segments differ in their
self-response rates via CQA without the presence of mailed materials with exposure to the
partnership and communications campaign and word-of-mouth as the only means of selfresponding.
C. Interventions with the 2020 Census
Name of solution/system/process: Content and Forms Design IPT
Explicit intervention requested
 Develop experimental mailing materials and questionnaires
 Assign form types to the newly developed materials.
Estimated impact: Minimal impact on the 2020 Census
Name of solution/system/process: National Processing Center (NPC)
Explicit intervention requested
 Printing of mailing materials
 Addressing materials
 Assembling mail packages
 Sending mail packages
Estimated impact: Minimal impact on the 2020 Census

20

Optimizing Self-Response in the 2020 Census Experiment, Version 1.5

Name of solution/system/process: Forms, Printing and Distribution
Explicit intervention requested
 Printing of mailing materials
 Assembling mail packages
Estimated impact: Minimal impact on the 2020 Census
Name of solution/system/process: CaRDS
Explicit intervention requested:
 Create the sample delivery file for the experiment
Estimated impact: Minimal impact on the 2020 Census
Name of solution/system/process: ECaSE-OCS
Explicit intervention requested
 Ingest the sample delivery file for the experiment
 Create workloads for each contact
 Send created workloads to NPC
Estimated impact: Minimal impact on the 2020 Census
Name of solution/system/process: Paper Data Capture
Explicit intervention requested:
 Receive and process experimental questionnaires
Estimated impact: Minimal impact on 2020 Census
Name of solution/system/process: iCADE
Explicit intervention requested:
 Data capture of experimental questionnaires
Estimated impact: Minimal impact on 2020 Census
Name of solution/system/process: Response Processing Operation
Explicit intervention requested:
 Process responses from experimental questionnaires
Estimated impact: Minimal impact on 2020 Census
D. Implications for 2030 Census Design Decisions and Future Research and Testing
This is an evaluation of the optimization of self-response in the 2020 Census. However, results
from this experiment can potentially be used to improve or enhance the strategies for
encouraging and motivating self-responses during the research and testing phase of the 2030
Census program and in the 2030 Census.

21

Optimizing Self-Response in the 2020 Census Experiment, Version 1.5

VI.

Data Requirements

Data File/Report

Source

Purpose

2020 Census Decennial
Response File

Census Data Lake

Analysis

Expected
Delivery Date
Fall 2020

VII. Risks
1. This experiment relies on NPC to assemble and mail packages. If NPC has commitments
to other surveys during this experiment’s mailout time, then the mail packages may not
be sent at the same time as the production 2020 Census materials.
2. If the data are not available in the Census Data Lake, then the analysis report cannot be
completed.
3. If sufficient funds are not granted for this experiment, than the scope of the experiment
may be reduced.
4. If the 2010 Census materials response rates are significantly higher than the response
rates from the 2020 Census materials, then further investigation would be warranted to
determine next steps.
5. Other Census Bureau groups will be sending additional mailers. It is possible that these
mailings will overlap with the mailings in this experiment. The minimum sample size is
being increased by twenty percent to account for possible overlap. If the overlap is not
random and significant, then the ability to make adequate conclusions about the effect of
the planned treatments may be negatively affected.

VIII. Limitations
1. Because of laws passed or updated after the 2010 Census, such as the Privacy Act and the
Cybersecurity Act, we are unable to use the exact same letters and postcard that were sent
in the 2010 Census.
2. There is an independence limitation since the 2020 Census communications campaign
will be advertising that the census can be completed online. It is virtually impossible to
create a “2010 no-internet environment” even though we will try to control for it by not
mentioning the internet option in the mailing materials.
3. Because of the fact that housing units within close proximity of each other may fall into
different categories (i.e., treatment versus control), it is difficult to control for the
possibility of communication between treatment and control groups. However, we
believe the likelihood, and therefore impact, of such communication is low.
4. When measuring the effects of withholding mailings on Internet and CQA response rates,
we will lack a comparison to a panel in which conditions are inverted, i.e., where there
are households that received mailed materials but were not exposed to the advertising
campaign.
5. Treatment E5 (i.e. No Mailout) will be reliant on non-ID procedures for internet and
CQA responses, so we will be unable to disentangle the effect on response rates because
22

Optimizing Self-Response in the 2020 Census Experiment, Version 1.5

of a non-ID environment from the effect on response rates because of offering no mail.
For example, we will be unable to measure whether non-ID procedures were more
burdensome than entering an ID, thereby decreasing internet and CQA response rates in
the treatment group.
6. Because of the fact that the URL and phone numbers used by households that respond by
the Internet or phone in the No Mailout panel will be identical to those used by control
households, we cannot tease out which component of the communication campaign drove
the internet or phone response (e.g. TV ad versus, print ad, versus a partnership specialist
material or event).

IX.

Issues That Need to be Resolved

X.

Division Responsibilities

Division or Office
Decennial Statistical Studies
Division

XI.

Responsibilities
 Plan and manage the experiment
 Design the panels
 Select the sample
 Monitor the results
 Analyze the data
 Write and release the report

Milestone Schedule

Experimental Milestone
Select Sample
2020 Census Self-Response
Receive, Verify, and Validate Data for Optimizing Self-Response
Experiment

Date
Winter 2019
March – August
2020
September 2020

Distribute Initial Draft of the Optimizing Self-Response Experiment
Report to the Decennial Research Objectives and Methods (DROM)
Working Group for Pre-Briefing Review

January 2021

Decennial Census Communications Office (DCCO) Staff Formally
Release the FINAL Optimizing Self-Response Experiment Report in the
2020 Memorandum Series

June 2021

23

Optimizing Self-Response in the 2020 Census Experiment, Version 1.5

XII. Review/Approval Table
Role

Approval Date

Primary Author’s Division Chief (or designee)

09/17/2018

Decennial Census Management Division ADC for Nonresponse,
Evaluations, and Experiments

03/20/2019

Decennial Research Objectives and Methods (DROM) Working Group

03/20/2019

Decennial Census Communications Office

06/03/2019

XIII. Document Revision and Version Control History
Version/Editor
0.1
0.2
1.0
1.2
1.3

Date
08/15/2018
09/17/2018
09/21/2018
03/8/2019
03/29/2019

1.4

06/03/2019

1.5

08/07/2019

Revision Description
Draft sent for team review
Draft sent for division chief review
Initial draft for DROM review
Second draft for DROM review
Final study plan submitted
Incorporated comments from Decennial Census
Communications Office
Updated sample sizes and corresponding text

XIV. Glossary of Acronyms
Acronym
ACS
ADC
DROM
FCC
IPT
LRS
MAFID
OSR
PDB
TEA

Definition
American Community Survey
Assistant Division Chief
Decennial Research Objectives and Methods
Working Group
Federal Communications Commission
Integrated Project Team
Low Response Score
Master Address File Identification
Optimizing Self-Response
Planning Database
Type of Enumeration Area

24

Optimizing Self-Response in the 2020 Census Experiment, Version 1.5

XV. References
Bentley, M., (2008), “Specifications for Bilingual Form Distribution in the 2010 Census (Phase
1),” U.S. Census Bureau: Decennial Statistical Studies Division 2010 Decennial Census
Memorandum Series #B-04. December 8, 2008.
Bentley, M., Reiser, C., Barron, S., Hill, J.M., Meier, A., Pape, T., (2014), “2020 Research &
Testing Program Research Results Report: 2012 Census Test Contact Strategy Results;
Optimizing Self Response (4.103),” U.S. Census Bureau: Decennial Statistical Studies
Division 2020 Decennial Census R&T Memorandum Series #E-04. November 4, 2014.
Bentley, M. and Rothhaas, C., (2015), “2020 Research & Testing Program Research Results
Report: 2014 Census Test Results; Optimizing Self Response (4.103),” U.S. Census
Bureau. 2020 Census Program Internal Memorandum Series: 2016.10.i. June 19, 2016.
Coombs, J., Lestina, F., and Phelan, J. (2017). “2020 Research & Testing Program Research
Results Report: 2016 Census Test Optimizing Self-Response, Language Services, Race
and Ethnicity, and Relationship Experiment Analysis Report,” U.S. Census Bureau. 2020
Census Program Internal Memorandum Series: 2017.22.i. September 8, 2017 .
Cernat, A. and Lynn, P., (2014), “The role of email addresses and email contact in encouraging
web response in a mixed mode design”. University of Essex Institute for Social and
Economic Research. Retrieved March 18, 2015 from
https://www.understandingsociety.ac.uk/research/publications/workingpaper/understanding-society/2014-10.pdf
Joshipura, M., (2008), “2005 ACS Respondent Characteristics Evaluation,” U.S. Census Bureau:
Decennial Statistical Studies Division. #ACS-RE-2. September 15, 2008.
Letourneau, E. (2012). “2010 Census Assessment Mail Response/Return Rates Assessment,”
U.S. Census Bureau: 2010 Census Planning Memoranda Series. Available at:
https://www.census.gov/2010census/pdf/2010_Census_Mail_Response_Return_Rates_A
ssessment.pdf.
Mathews, K., (2015), “Sample Design Specifications for the 2015 National Content Test,” U.S.
Census Bureau. Decennial Statistical Studies Division 2020 Decennial Census R&T
Memorandum Series #R-11. November 18, 2015.
Mathews, K. and Phelan, J. (2018). “Determining Internet Choice Areas for the 2020 Census,”
U.S. Census Bureau (DRAFT).

25

Optimizing Self-Response in the 2020 Census Experiment, Version 1.5

Mathews, K. and Rothhaas, C., (2015), “Sample Design Specifications for the 2015 Census
Test,” U.S. Census Bureau: Decennial Statistical Studies Division 2020 Decennial
Census R&T Memorandum Series #E-05. April 29, 2015.
Nichols, E., Horwitz, R., and Tancreto, J., (2014), “Do hard-to-interview groups self-respond
more when offered an Internet reporting option?” U.S. Census Bureau : 2014 American
Community Survey Research and Evaluation Report Memorandum Series #ACS14-RER26. October 21, 2014.
Phelan, J. (2016). “2020 Research & Testing Program Research Results Report: 2015 National
Content Test Optimizing Self-Response Report,” U.S. Census Bureau. 2020 Census
Program Internal Memorandum Series: 2016.57.i. November 22, 2016.
U.S. Census Bureau (in preparation). 2020 Census Predictive Models and Audience
Segmentation Report.

26

Optimizing Self-Response in the 2020 Census Experiment, Version 1.5

XVI. Appendix A
The formula used to calculate the minimum sample size necessary for the desired comparisons is

where
= minimum sample size
= minimum detectible difference

* = alpha level adjusted for multiple comparisons (Bonferroni)
Z*/2 = critical value for set alpha level assuming a two-sided test
= critical value for set beta level
Z
p1
= proportion for group 1
p2
= proportion for group 2
deff = design effect due to unequal weighting
Wang, H. and Chow, S. (2007). “Sample Size Calculation for Comparing Proportions,” Wiley
Encyclopedia of Clinical Trials (eds R.B. D’Agostino, L. Sullivan, and J. Massaro).
n

 =

0.014
=
0.014
*
Z*/2 = 0.8416212
n with deff = 10,396.8503
Z = 0.841621
p1 =
0.57 n without deff = 5,941.0573
p2 =
0.54
deff =
1.75







The sample sizes for the Optimization of Self-Response Experiment and the national
sample of the Extending the Census Environment to the Mailing Materials Experiment
were calculated simultaneously. The samples share a production language control, where
housing units are given mail materials as specified by production definitions, and a
bilingual control, where housing units are given bilingual mail materials regardless of the
production definitions and will be selected simultaneously.
The value of p1 is the expected 2020 Census response rate after six weeks, which is
between the minimum and average expected 2020 Census response rate.
The estimated design effect is 1.75, based on 2015 NCT data evaluation of the variable
analysis_response. The OSR sample for 2015 NCT is the most complex, and has a deff =
1.2351. The design effect of the entire 2015 NCT is 1.9513.
The values of n found here are the numbers needed in both group 1 and group 2 to detect
a 3 percentage point difference. For example, if only two groups were to be compared,
we would need 20,796 housing units with the given parameters. The sample for the
Optimization of Self-Response Experiment includes 11 groups that will be compared:
Panels C1 and C2, contain two groups each with the sample being split between Internet
27

Optimizing Self-Response in the 2020 Census Experiment, Version 1.5

First and Internet Choice. Panels Panel E1 and E2 contain one group with the sample split
between Internet First and Internet Choice. Panel E3 has two groups with the sample only
in Internet Choice while Panel E4 has two groups with the panel only in Internet First.
Panel E5 has three groups where the sample is two-thirds Internet First and one-third
Internet Choice. For the 11 groups, the total sample size is 98,781.

28