2020 Census Evaluation- The Undercount of Young Children Study Plan

The memorandum and attached document(s) was prepared for Census Bureau internal use. If
you have any questions regarding the use or dissemination of the information, please contact
the Stakeholder Relations Staff at [email protected].

2020 CENSUS PROGRAM INTERNAL MEMORANDUM SERIES: 2019.18.i
Date:

May 1, 2019

MEMORANDUM FOR: The Record
From:

Deborah M. Stempowski (signed May 1, 2019)
Chief, Decennial Census Management Division

Subject:

2020 Census Evaluation: The Undercount of Young Children: A Qualitative
Evaluation of Census Materials and Operations Study Plan

Contact:

Jennifer Reichert
Decennial Census Management Division
301-763-4298
[email protected]

This memorandum releases the final version of the 2020 Census Evaluation: The Undercount of Young
Children: A Qualitative Evaluation of Census Materials and Operations Study Plan, which is part of the
2020 Census Program for Evaluations and Experiments (CPEX). For specific content related questions,
you may also contact the authors:
Joanne Pascale
Center for Behavioral Science Methods
301-763-4920
[email protected]

Jessica Ellen Graber
Center for Behavioral Science Methods
301-763-6550
[email protected]

Anna Sandoval Giron
Center for Behavioral Science Methods
301-763-3575
[email protected]

Alda Rivas
Center for Behavioral Science Methods
301-763-3396
[email protected]

Shelley B. Feuer
Center for Behavioral Science Methods
301-763-0873
[email protected]

census.gov

2020 Census Evaluation
The Undercount of Young Children:
A Qualitative Evaluation of Census
Materials and Operations Study Plan
Part Two (CPEX): FINAL
Joanne Pascale
Jessica Graber
Shelley Feuer
Anna Sandoval Girón
Alda Rivas
Center for Behavioral Science Methods
May 1, 2019
Version 2.1.

Page intentionally left blank.

Undercount of Young Children Qualitative Evaluation, Version 2.1

Table of Contents
I.

Introduction ......................................................................................................................... 1

II.

Background ......................................................................................................................... 2

III.

Assumptions........................................................................................................................ 3

IV.

Research Questions ............................................................................................................. 3

V.

Methodology ....................................................................................................................... 3

VI.

Data Requirements ............................................................................................................ 10

VII.

Risks.................................................................................................................................. 10

VIII.

Limitations ........................................................................................................................ 10

IX.

Issues That Need to be Resolved ...................................................................................... 11

X.

Division Responsibilities .................................................................................................. 11

XI.

Milestone Schedule ........................................................................................................... 11

XII.

Review/Approval Table .................................................................................................... 12

XIII.

Document Revision and Version Control History ............................................................ 12

XIV. Glossary of Acronyms ...................................................................................................... 12
XV.

References ......................................................................................................................... 13

Undercount of Young Children Qualitative Evaluation, Version 2.1

Page intentionally left blank.

Undercount of Young Children Qualitative Evaluation, Version 2.1

I.

Introduction

There is well-established literature indicating that young children (under age 5) are systematically
undercounted in censuses and surveys around the world, and the U.S. decennial census and
American Community Survey (ACS) are no exception (O’Hare, 2015). In the 2010 Census, for
instance, there was a net overcount for the population as a whole of 0.1 percent, but for young
children there was a net undercount of 4.6 percent. In the 2015 ACS, the undercount rate was much
worse, at 13 percent. The magnitude of the problem is exacerbated by related evidence that the
undercount varies by demographic characteristics. The 4.6 percent undercount in 2010 was for all
children under 5; among non-Hispanic Whites the undercount was 2.7 percent, while for Blacks it
was 6.3 percent, and for Hispanics it was 6.5 percent. Furthermore, these figures are for the net
undercount; the omission rate of young children in 2010 was more than 10 percent (Griffin and
Konicki, 2017).
While research to date has established the magnitude of the problem and uncovered some aspects
of the types of households and individuals most affected, what is not known is why respondents
fail to include young children on the initial census form. That is, questions of whether and how the
census and survey forms, question wording, interviewer instructions, and other procedures
contribute to the undercount, and whether modifications to these methods could reduce the
undercount, have not yet been explored. One key component and a potential source of
measurement error contributing to the undercount is the methodology used to gather the
“household roster”—that is, the names of individuals who “live or stay” at the address. One person
(known as the “household respondent” or HHR) completes the census on behalf of all household
members, and the relationship between the HHR and these other individuals could be an important
factor in whether or not they list them on the form. For example, in a large, complex household
with several distinct families, not all household members may be closely related to the person who
completes the census on behalf of all of them. Thus, a HHR may list their own children, but not
those of a distant relative whose child goes back and forth among other relatives’ houses.
In this evaluation, we aim to explore reasons for the undercount using focus groups and cognitive
interviews. Similar methods will be employed in a pre-census sister study (“The Undercount of
Young Children: A Qualitative Evaluation of Census Materials and Operations Part 1”) starting in
the spring of 2019 (Pascale et al, 2019), and results will be used to refine the protocol for this
CPEX study. We will also rely on related literature on complex households (Schwede, 2006;
2013), “family boundary ambiguity” (Brown and Manning, 2009), rostering procedures
(Tourangeau et al, 1997; Kephart and Krosnick, 2018), ethnographic analysis (de la Puente, 1992),
and studies from the 1990s “Living Situation Survey” (Research Triangle Institute, 1992). The
overarching objective is to explore in more depth where the existing roster questions and
procedures are failing and how, specifically, to improve them. Probing will explore individuals’
interpretation of the current wording of the roster questions and their rationale for including or
excluding certain individuals. Vignettes will be used to examine how respondents would complete
the roster questionnaire under various complicated and known-to-be-problematic situations, and
follow-up probes will explore why respondents included or excluded particular individuals.
Results could feed into more targeted research into improvements to the census forms and other
1

Undercount of Young Children Qualitative Evaluation, Version 2.1

aspects of data collection in 2030. Results could also inform ongoing surveys that involve a asking
a single household member to list all household members.
II.

Background

In 2014, the Census Bureau published the final report from a task force set up to examine the
undercount of young children (U.S. Census Bureau, 2014). In 2015, the Census Bureau established
the Undercount of Young Children Work Group (UYCWG), which began tackling this problem
in a systematic way and produced several official memoranda documenting their findings over the
course of two years (see Appendix A). Generally, results identified some key factors of the
undercount associated with household complexity and composition, and the relationship between
the HHR and the omitted child. For example, children are more likely to be undercounted if:
• The child lives in a “complex household,” which includes multigenerational households,
households with nonrelatives, nuclear families (married couples and their biological
children) living with other more distant relatives, and blended families.
• The child is not the biological or adoptive parent of the child, and may be a grandchild,
distant relative or nonrelative.
• The child’s mother is under age 25.
While the work of the UYCWG was not technically part of the 2012-2018 Research and Testing
endeavors, research on this topic was a priority to the Census Bureau in the years leading up to
2020. Indeed, work of the team was highlighted in the Census Bureau Director’s blog in July 2018
Below is an excerpt:
“Over the decades, the Census Bureau has vastly improved its procedures to reduce or
even eliminate occurrences of both overcounted and undercounted persons. As in the
past, we are taking the lessons learned from previous surveys and studies to ensure a
complete and accurate 2020 Census. In particular for 2020, we are well aware of and
focused on the problem of undercounting young children ages 0 to 4 — a critical issue
highlighted in a recent report by the Annie E. Casey Foundation.” (Jarmin, 2018).
Conducting this evaluation will provide further in-depth understanding of the undercount of young
children and provide insights into possible solutions and strategies to reduce the undercount.

2

Undercount of Young Children Qualitative Evaluation, Version 2.1

III.

Assumptions
•
•

IV.

We assume there will be funding for this project, as described below in the methodology
section.
We assume that the data collected by other divisions will be made available to complete
this project.
Research Questions

1. What aspects of census materials and operations (forms, question wording and format,
interviewer instructions, mode of data collection, proxy response, etc.) contribute to children
under 5 years old being erroneously omitted from the census?
2. Why are young children from some racial/ethnic groups more likely to be omitted than others?
3. What kind of modifications to census operations could reduce erroneous omissions of young
children?
4. Do the factors associated with the omission of young children have any implications for other
age groups, and for households where multiple household members are omitted, in addition to
young children?
5. What can we learn about messaging with regard to “confusion” versus “concealment” as
factors contributing to the omission of young children?
a. Confusion: Are there aspects of data collection that could feed in to outreach campaigns
(e.g., notification that ALL household members should be counted, not just adults)?
b. Concealment: Do respondents express any privacy and/or confidentiality concerns with
regard to young children in particular? That is, is there any evidence of deliberate
concealment that could be addressed through a public relations campaign with, for
example, “trusted messengers”?
V.

Methodology

A. Design
1. Overview
We plan to conduct qualitative research with individuals recruited from two different types of
sample (described below) who may have erroneously omitted young children from their census
form. Racial/ethnic minorities will be oversampled across both types of sample. Once identified,
individuals will be recruited for either a cognitive interview or a focus group. Both cognitive
testing and focus groups will explore the question wording and other aspects of the data collection
operations, such as respondent comprehension, reasons for selecting their answers, judgments and
interpretations of the instructions, and so on. The cognitive interviews will use one-on-one indepth probing to explore individual-level reactions to the census materials, while the focus groups
will use a more open group discussion format to probe the same types of issues. We expect these
methods to help answer research questions #1 and #3. We also expect that by recruiting on
racial/ethnic groups we will have sufficient data to compare and contrast reporting error across a
3

Undercount of Young Children Qualitative Evaluation, Version 2.1

broad spectrum of demographic characteristics to address research question #2. Similarly, while
we are prioritizing undercounting young children, we expect that many of our respondents will
live in households with older individuals as well, and that results will help answer research
question #4.
In addition to issues related to details of the questionnaire, the focus groups will cover more broad
topics such as respondents’ perceptions of the kinds of individuals who “should” be listed on the
form as “living or staying” at the household, how it is decided which household member will fill
out the form, and topics that could feed into future messaging campaigns. We expect these results
to help answer research question #5.
In terms of scope, the census operations we believe are most relevant and appropriate for this type
of evaluation are:
a. Internet Self-Response (ISR)
b. Self-administered paper form
c. Nonresponse Followup (NRFU)
While the content is fairly similar across these operations, we would like to assess any evidence
of mode effects (e.g., between internet and paper), and interviewer effects (e.g., by comparing both
self-response modes to the NRFU mode).
2. Sample
The population of individuals who erroneously omit young children from the census is elusive.
We have some empirical evidence on this population from various census follow-up operations
and post-enumeration surveys (PES), but in some cases, the same reporting errors that plague the
initial census also affect these follow-up studies. Thus, while we can examine the characteristics
of individuals and households who are found to add young children to the census via PES-type
operations, we do not know how those characteristics compare to the “true” population of
households where young children are erroneously omitted.
We aim to leverage the existing body of knowledge on the undercount of young children by using
a two-pronged approach to identify sample for qualitative research. In Sample A we draw sample
from the PES operation—that is, households where at least one young child was found in the PES
but not in the initial census. In Sample B we draw sample using administrative records (adrecs)
linked to completed census forms from the 2020 Census in order to identify households where at
least one child was indicated to live in the household in the adrecs but not in the census.
Both approaches, described in more detail below, have their strengths and weaknesses. While
neither method can be said to be completely representative of the true population of households
where young children are undercounted, we believe we can learn from interviews with individuals
identified in each approach. Some of the findings from each method could overlap, and some of
findings could be unique to each method. That is, the two methods could complement each other
nicely. We also note that while the focus of this research is on young children, past research
indicates that in households where at least one young child was erroneously omitted, they were
not the only individuals missing from the original census form; indeed in 68 percent of cases at
4

Undercount of Young Children Qualitative Evaluation, Version 2.1

least one other household member was missing on the original census (and in 46 percent of cases
the whole household was missed) (U.S. Census Bureau, 2017). Thus there is some potential that
any mechanisms that may be associated with the omission of young children may also affect the
omission of other household members. So while our sampling strategy will target households
where young children were omitted, our protocols will address omissions of other individuals as
well.
Sample A: Post-Enumeration Survey (PES)
To identify sample from the Post-Enumeration Survey (PES), we plan to build on the extensive
research conducted over the past few years and documented in two memos (Memorandum
2017.04: Undercount of Children – Coverage Measurement and Memorandum 2017.05:
Undercount of Young Children – Coverage Followup). These studies provide a very useful
roadmap for identifying cases likely to yield valuable results from qualitative research. For
example:
• The PES analysis starts with “P-sample nonmatches”—people and households included in
the PES sample that could not be matched to a household/person in the original census.
The sample will include “true omissions” (where a child was listed in the PES but not in
the original census) along with cases where the child was listed on the original census but
without sufficient detail to be matched to the corresponding PES household, and/or the
child was listed in different places in the PES and the original census. For purposes of
qualitative testing, we plan to narrow down the initial PES sample by focusing on PES
households in which a child did not match to the census, but other people in the PES did.
We may also use indirect evidence (e.g., person-level imputation in the matched census
household) and screener questions to identify only the very-likely true omissions. This
could help us hone in on aspects of the census forms, question wording, instructions, etc.
that may have led the respondent to fail to omit the child altogether in the initial
enumeration.
• The PES analysis found that in almost 68 percent of cases where a young child was
identified in the PES but could not be matched to the original census form, at least one
other household member could also not be matched. This suggests that issues related to
omissions of young children could affect omissions of other age groups as well.
• The PES results indicated that in about 46 percent of nonmatched cases, ALL PES
household members could not be matched to the initial census. This indicates that both
within-household and whole-household errors contribute about equally to the undercount.
In terms of strategy, it suggests that it would be valuable to attempt to sample households
from the initial census where the corresponding PES household contained no individuals
that were rostered in the initial census. This would give us a chance of interviewing
individuals in households that were missed altogether in the initial census and all NRFU
operations. Protocols for this sample would emphasize participation in the census in
general, and focus less on the individual rostering of particular household members.
• As noted above, research has identified certain characteristics of households where young
children tend to be omitted. For example, those more at risk are children who are the
grandchildren, other relatives, and nonrelatives of the householder; non-Whites, and those
5

Undercount of Young Children Qualitative Evaluation, Version 2.1

living in renter-occupied units. However, authors of the memos noted above also state
“Households that fit the description of being ‘easier-to-enumerate’ still include errors that
add substantially to this undercount problem. Despite having some of the lowest nonmatch
rates, the characteristics that involve the majority of nonmatches include White children,
non-Hispanic children, biological children, and nonmovers. We will not resolve the
undercount of young children without addressing these populations.” This suggests that we
sample both “at risk” and “easy to enumerate” cases.
We expect there would be sufficient sample size in the PES for selective subsampling. For
example, in the 2010 CCM study the authors state, “The CCM was unable to match 3,058 P-sample
young children to a person record in the 2010 Census.” Assuming a similar overall sample size in
the 2020 PES would yield around 3,000 cases of households where young children were not
matched, which should be a sufficient base from which to subsample.
We recognize that cases in the PES may have been contacted multiple times and we are sensitive
to the associated respondent burden issues. As a mitigating factor, we will be offering monetary
incentives for the qualitative research, and we will be conducting the research with a relatively
small number of individuals (less than 200 cases). We also note that there is some precedent for
post-PES-type studies. For example, in 2000 there was a “measurement error reinterview” that
followed the Accuracy and Coverage Evaluation” (Mulry, 2006).
Sample B: AdRecs
Our Census Bureau colleagues have worked out a method of linking adrecs to the 2010 Census to
identify households where individuals appear to be missing from the census form (Mule, Keller
and Konicki, 2019). In the pre-census sister study noted above (“Part 1”), we plan to pilot this
basic approach to identify where young children, in particular, were erroneously omitted from the
census based on the 2018 End-to-End Census Test. We then plan to conduct qualitative interviews
with Rhode Island sample in the summer/fall of 2019. We expect to make any needed refinements
to the sampling protocol and then employ the same basic approach; linking adrecs to 2020 Census
records in order to target households who erroneously omitted young children on their form.
Site Selection
For practical purposes (i.e.: budgeting and scheduling), we envision conducting data collection in
only three or four geographic areas. Thus, the geographic concentration of households identified
as omitting at least one young child on the census form via the PES and adrec methods may dictate
where we collect data. For example, if there are only a few large metro areas where there is a
sizeable caseload of households where young children were omitted; those would be our data
collection sites. If there is a sufficient caseload in several geographic areas, we plan to select sites
based on pragmatic considerations (proximity to Suitland). If additional sites are possible, we will
turn to research by our Census Bureau colleagues, who used an enhanced demographic analysis
technique to produce county-level estimates of net coverage error of young children (King, Ihrke
and Jensen, 2018). For Part 1 of this study (planned for spring/fall of 2019), we consulted with
Eric Jensen and Heather King to discuss ways to leverage their analysis for site selection. We
6

Undercount of Young Children Qualitative Evaluation, Version 2.1

sought to identify areas of the country that appeared to be “hot spots” of the undercount. We plan
to conduct the same type of analysis for Part 2 of this evaluation. Their analysis shows a
concentration of high-undercount counties contiguous with each other in these areas (see Figure
1, from their report, reproduced below):
• Mississippi, along the river from about Natchez up to Tunica.
• Texas (southern tip), including Brownsville.
• Southeast Virginia.
• Southwest West Virginia.
• Southeast Kentucky.

Figure 1. Subnational demographic analysis estimates of net coverage error by county.
Reproduced from King et al. (2018).
Their analysis also integrated demographic and housing characteristics with the undercount data
to produce a map (see Figure 2 from the report, reproduced below). Specifically, they used “spatial
and cluster analysis to highlight patterns in the geographic distribution of coverage errors for
young children by demographic and housing characteristics….to group together demographically
similar counties into mutually exclusive groups. We identified seven clusters:
1. Majority Black and White
2. Majority American Indian/Alaska Native
3. High Hispanic proportion
4. Immigrant destinations
5. Prison and military counties
6. Average characteristics
7

Undercount of Young Children Qualitative Evaluation, Version 2.1

7. Mostly White.”

Figure 2. County clusters formed from demographic and housing characteristics. Reproduced from
King et al. (2008).
These findings, examined in tandem with the results on contiguous counties with high undercounts
(Figure 1), suggest two parts of the country (outside the DC area) that may represent unique
patterns of misreporting:
• The Mississippi delta (Cluster 1/Majority Black and White), and
• The southern tip of Texas (Cluster 3/High Hispanic proportion).
Thus, these could represent good candidates of sites for data collection, depending on other data
from the PES and adrecs samples.
We make one final note on timing. We recognize that the earliest we may be able to be in the field
with this research is spring 2021. This risks changes in household composition between “Time 1”
(the initial census) and “Time 2” (our qualitative research). It also risks that respondents may have
little recollection of what they answered in Time 1. However, our goal in this study is not
necessarily to reconcile the household roster at Time 1 with Time 2. Rather it is to flag households
either at risk of omitting young children or who actually did omit young children, and then probe
8

Undercount of Young Children Qualitative Evaluation, Version 2.1

them on the reasons for those omissions. To that end, the key factor is that the household was
flagged as having some type of measurement error in the initial census with regard to the household
roster. So while these above two concerns may not be trivial, the qualitative research may actually
benefit by data collection taking place in the months that roughly mimic the peak times of data
collection for the actual census (March/April). To the extent that respondents completing the
census in March have difficulty predicting who will be “living or staying” in the household on
April 1, and respondents completing the census after April 1 have difficulty remembering who was
“living or staying” there on April 1, conducting our research in these months could be quite
informative.
3. Focus Groups
We plan to conduct 12 focus groups to examine where the existing roster questions and procedures
are failing with regard to young children and how, specifically, to improve them. We will base our
protocol on the moderator guide being used in the “Pre-Census Part 1” study noted above (see
Appendix B), which includes the following sub-topics to be covered in the focus groups:
• Past experience with and perceptions of the decennial census.
• Within-household negotiation over who completes the census on behalf of the household
• An exercise in completing the household roster, followed by a question-by-question group
discussion on interpretation of the questions, instructions, etc., and the rationale for
responses in the rostering exercise.
• Knowledge of the census.
• Vignettes to examine how respondents would complete the roster questionnaire under
various complicated and known-to-be-problematic situations, and follow-up group
discussion on why participants would include or exclude particular individuals.
• Messaging to explore factors that would encourage or discourage participation
The Pre-Census Part 1 study is scheduled for the spring/fall of 2019 and results will be used to
refine and enhance the protocol for this study.
4. Cognitive Interviews
We plan to augment the focus groups with at least 90 cognitive interviews (30 in each operation –
ISR, paper and NRFU) in order to explore many of the same basic issues but with a one-on-one
more in-depth method. Probing will explore individuals’ interpretation of the current wording of
the roster questions, and their rationale for including or excluding certain individuals. As with the
focus groups, the protocol (see Appendix C) will be based on the “Pre-Census Part 1” study. The
interviews will be broken up into “rounds” as needed. That is, after the first handful of interviews
is conducted, results will be assessed for any fatal flaws in the protocols. Methods will be adjusted
as needed for the next round. Results from each round of testing will be fed into modifications to
the protocols for the subsequent round until, ideally, “saturation” is reached. This is the point at
which testing with additional cases using the same methodology no longer yields new findings. If
this point is reached and there are still remaining cases that have been budgeted for, and if results
provide evidence for specific modifications to the questions, interviewer procedures, etc., then
testing of those modifications could be conducted in a verification round in an attempt to assess
9

Undercount of Young Children Qualitative Evaluation, Version 2.1

any reductions in measurement error. Findings could inform testing of alternative questions and
probes as a starting point for the 2030 Census, and for redesigns in ongoing surveys such as the
ACS.
B. Interventions with the 2020 Census
There will be no interventions during data collection. Some preparatory work will be conducted in
the spring/fall of 2019 for the Pre-Census Part 1 study, and refinements to the CPEX/Part 2
sampling methodology will be incorporated as appropriate after the 2020 data collection is
complete.
C. Implications for 2030 Census Design Decisions and Future Research and Testing
The research could serve as a starting point for testing of the 2030 Census, such as redesigned
instruments, forms, instructions and enumerator training. Findings could also inform testing on
reducing coverage error in ongoing surveys such as the ACS.
VI.

Data Requirements

The raw data for analysis will consist of sample from the PES and adrecs, transcripts from focus
groups, audio recordings of the cognitive interviews and summaries of both. We plan to have a
transcription company transcribe the focus groups and CBSM staff will produce all the other raw
data, and the analysis.
Data File/Report

Source

Purpose

Expected
Delivery Date

Audio recordings of focus
groups
Focus group transcripts

CBSM staff conducting
focus groups
Transcription company

Summer 2021

Cognitive interview summaries

CBSM staff conducting
interviews

PES and adrec sample file

Decennial

Capture a record of the
discussion for later analysis
Produce transcripts of focus
group audio recordings
Written summary of questionby-question discussion of
census forms, instructions and
other aspects of data collection
To identify sample for focus
groups and cognitive interviews

VII.

Summer 2021
Fall 2021

Early 2021

Risks

Risks will be mitigated by a pilot test (begun in early spring 2019 and expected to finish late spring
2019), and the Pre-Census Part 1 version of this study.
VIII. Limitations
A. One limitation, as with all qualitative research, is that we may not be able to capture the full
range and diversity of respondents who underreport young children in our sample, and thus we
10

Undercount of Young Children Qualitative Evaluation, Version 2.1

may miss certain perspectives. Our participant recruitment strategy, although not
representative, seeks to find respondents that reflect those populations at-risk of undercount.
We expect that the one-on-one cognitive interviews (where we can explore in depth the
reporting issues) combined with focus groups (where we can explore some of the same issues,
but with larger numbers of individuals) cast a wide enough net so that our findings do not miss
any major sector of the population at-risk for under-reporting young children.
B. Another limitation could be interview mode. We plan to conduct testing with the paper form,
the internet self-response (ISR) form, and the Nonresponse Followup (NRFU) instrument.
Given the complexities and unknowns around the testing windows of these modes, we may or
may not be able to truly simulate the experience of respondents in these operations.
IX. Issues That Need to be Resolved
A. Complete pilot test (spring 2019)
B. Complete Pre-Census Part 1 study in summer/fall of 2019
X.

Division Responsibilities

Division or Office
Research and
Methodology/Center for
Behavioral Science Methods

Transcription service
Demographic
Programs/Population
Division
Decennial Statistical Studies
Division

XI.

Responsibilities
• Seek funding and staff support
• Develop detailed study plan, protocols and schedule
• Conduct focus groups and cognitive interviews
• Summarize focus group transcripts and cognitive
interviews
• Analyze focus groups and cognitive interviews
• Write draft and final reports
• Transcribe focus group audio recordings
• Review all study plan materials
• Observe cognitive interviews and focus groups
•
•
•

Work with CBSM staff to identify PES and adrecs sample
Review all study plan materials
Observe cognitive interviews and focus groups

Milestone Schedule

MILESTONES
PREPARATION FOR DATA COLLECTION
Receive approval and funding from Decennial
Refine protocol for focus groups and cognitive interviews based on PreCensus Part 1 study
Obtain OMB clearance for main stage
Kick-off meeting
11

End Date
3/7/2019 (?)
Early 2020
Early 2020
Late 2020

Undercount of Young Children Qualitative Evaluation, Version 2.1

MILESTONES
DATA COLLECTION: SAMPLES A (PES) and B (AdRecs)
DATA REDUCTION, CODING, ANALYSIS AND REPORT WRITING
XII.

End Date
Spring thru
summer,
2021
Early 2022

Review/Approval Table

Role

Approval Date

Primary Author’s Division Chief (or designee)

3/19/2019

Decennial Census Management Division (DCMD) ADC for
Nonresponse, Evaluations, and Experiments

mm/dd/yyyy

Decennial Research Objectives and Methods (DROM) Working Group

3/7/2019

Decennial Census Communications Office (DCCO)

mm/dd/yyyy

XIII. Document Revision and Version Control History
Version/Editor
Date
Revision Description
1.0/Joanne Pascale April 2, 2019 Original Submission
2.1/Anna
Address comments and editorial review from Decennial.
April 30, 2019
Sandoval Girón

XIV. Glossary of Acronyms
Acronym
ADC
CI
DCCO
DROM
DSSD
EXC
FG
IPT
R&M
UYC

Definition
Assistant Division Chief
Cognitive Interview
Decennial Census Communications Office
Decennial Research Objectives and Methods
Working Group
Decennial Statistical Studies Division
Evaluations & Experiments Coordination Branch
Focus Group
Integrated Project Team
Research & Methodology Directorate
Undercount of Young Children

12

Undercount of Young Children Qualitative Evaluation, Version 2.1

XV. References
Brown, Susan L. and Wendy D. Manning. 2009. “Family Boundary Ambiguity and the
Measurement of Family Structure: The Significance of Cohabitation.” Demography. 46: 85.
https://doi.org/10.1353/dem.0.0043
De la Puente, Manuel. 1993. “Using Ethnography to Explain Why People Are Missed or
Erroneously Included by the Census: Evidence from Small Area Ethnographic Studies.”
https://www.census.gov/srd/papers/pdf/mdp9501.pdf
Griffin, Deborah and Scott Konicki. 2017. “Investigating the 2010 Undercount of Young
Children – A Comparison of Demographic, Social, and Economic Characteristics of Children by
Age.” 2020 CENSUS PROGRAM MEMORANDUM SERIES: 2017.14
https://www.census.gov/programs-surveys/decennial-census/2020-census/planningmanagement/memo-series/2020-memo-2017_14.html
Jarmin, Ron. 2018. “Improving Our Count of Young Children.” U.S. Census Bureau Director’s
Blog. https://www.census.gov/newsroom/blogs/director/2018/07/improving_our_count.html
Kephart, Kathleen and Jon Krosnick. 2018. “To List or Not to List? That is the Question: An
Examination of Existing Research on the Challenges and Best Practices of Household
Rostering.” Poster presented at the American Association for Public Opinion Annual Research
Conference, Denver, CO.
https://www.census.gov/content/dam/Census/newsroom/press-kits/2018/aapor/aapor-poster-hhrostering.pdf
King, Heather, David Ihrke and Eric Jensen. 2018. “Subnational Estimates of Net Coverage
Error for the Population Aged 0 to 4 in the 2010 Census.” Presented at the Annual Meeting of
the Population Association of America, Denver, CO, April 26-28, 2018.
https://www.census.gov/content/dam/Census/newsroom/press-kits/2018/paa/2018-paapresentation-net-coverage-error.pdf
Mule, Thomas, Andrew Keller and Scott Konicki. “Administrative Record Dual System
Estimation Study Plan.” Presented to the Decennial Research Objectives and Methods Working
Group. Spring, 2019.
Mulry, Mary. 2006 “Summary of Accuracy and Coverage Evaluation for Census 2000” SRD
Working Paper #2006-03
O’Hare, William. 2017. “Geographic Variation in 2010 U.S. Census Coverage Rates for Young
Children: A Look at Counties. International Journal of Social Science Studies Vol. 5, No. 9;
September 2017 ISSN 2324-8033 E-ISSN 2324-8041 Published by Redfame Publishing URL:
http://ijsss.redfame.com

13

Undercount of Young Children Qualitative Evaluation, Version 2.1

O’Hare, William. 2014. Historical Examination of Net Coverage Error for Children in the U.S.
Decennial Census: 1950 to 2010. Research and Methodology Directorate, Center for Survey
Measurement Study Series (Survey Methodology #2014-03). U.S. Census Bureau. Available
online at https://www.census.gov/srd/papers/pdf/ssm2014-03.pdf
O’Hare, William. 2014. Assessing Net Coverage Error for Young Children in the 2010 U.S.
Decennial Cen. Research and Methodology Directorate, Center for Survey Measurement Study
Series (Survey Methodology #2014-02). U.S. Census Bureau. Available online at
https://www.census.gov/srd/papers/pdf/ssm2014-02.pdf
Pascale, Joanne, Jessica Graber, Shelley Feuer, Anna Sandoval Girón, Alda Rivas. 2019. “The
Undercount of Young Children: A Qualitative Evaluation of Census Materials and Operations
Part 1.” Pre-Census Study Design Proposal.
Research Triangle Institute. 1992. “Living Situation Survey Report on Questionnaire Pretest
Activities.” https://wwwn.cdc.gov/qbank/report/LivingSituationReport.pdf
Schwede, L. (2003), “Complex Households and Relationships in the Decennial Studies and in
Ethnographic Studies of Six Race/Ethnic Groups.” Census 2000 Testing, Experimentation and
Evaluation Program Ethnographic Research Report. Suitland, MD: U.S. Census Bureau.
http://www.census.gov/pred/www/rpts/Complex%20Households%20Final%20Report.pdf
Schwede, L. (2006), “Who Lives Here? Complex Ethnic Households in America.” In Complex
Ethnic Households in America, edited by L. Schwede, R. Blumberg, and A. Chan. Lanham, MD:
Rowman & Littlefield Publishers, Inc., pages 280-320.
Schwede, L, Eric Jensen and Debbie Griffin. (2018). “Measuring Linkages Among Household
Complexity, Race/Ethnicity and the Undercount of Young Children in Censuses.” Presentation at
the American Association for Public Opinion Research.
Tourangeau, Roger, Gary Shapiro, Anne Kearney and Lawrence Ernst. 1997. “Who Lives Here?
Survey Undercoverage and Household Roster Questions.” Journal of Official Statistics, Vol. 13,
No. 1, 1997, pp. 1-18
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.451.3363&rep=rep1&type=pdf
U.S. Census Bureau. 2014. Final Task Force Report on the Undercount of Young Children.
https://www.census.gov/content/dam/Census/library/working-papers/2014/demo/2014undercount-children.pdf
U.S. Census Bureau. 2016. “Investigating the 2010 Undercount of Young Children – A New
Look at the 2010 Census Omission by Age.”
2020 CENSUS PROGRAM MEMORANDUM SERIES: 2017.04
https://www.census.gov/programs-surveys/decennial-census/2020-census/planningmanagement/memo-series/2020-memo-2016_09.html
14

Undercount of Young Children Qualitative Evaluation, Version 2.1

U.S. Census Bureau. 2017. “Investigating the 2010 Undercount of Young Children – Examining
Data Collected during Coverage Followup.”
2020 CENSUS PROGRAM MEMORANDUM SERIES: 2017.05

15

Undercount of Young Children Qualitative Evaluation, Version 2.1

Appendix A: Research Products from the Undercount of Young
Children Work Group
Nine published research reports in the public 2020 memo series.
https://www.census.gov/programs-surveys/decennial-census/2020-census/planningmanagement/memo-series.html
2016.08: Young Mothers
2016.09: Omissions by Age
2017.02: Demographic, Housing and Household Characteristics by Age
2017.04: Census Coverage Measurement (CCM)
2017.05: Coverage Follow-Up
2017.14: Demographic, Social and Economic Characteristics by Age
2017.15: Geographic Distribution
2017.16: ESRI Tapestry Segmentation
2018 Population Association of America paper/posters
https://www.census.gov/content/dam/Census/newsroom/press-kits/2018/paa/2018-paapresentation-net-coverage-error.pdf
https://www.census.gov/content/dam/Census/newsroom/press-kits/2018/paa/2018-paa-posterhispanic-children.pdf
2018 CARRA working paper
https://www.census.gov/content/dam/Census/library/working-papers/2018/adrm/carra-wp-201805.pdf
April 2016 Program Management Review presentation
https://www2.census.gov/census_2020/pmr_materials/2016-0412/PrePMR%20The%20Undercount%20of%20Young%20Children.pdf
Presentations to NAC
October 2015
https://www2.census.gov/cac/nac/meetings/2015-10/2015-Konicki.pdf
November 2016
https://www2.census.gov/cac/nac/meetings/2016-11/2016-konicki.pdf
November 2017
https://www2.census.gov/cac/nac/meetings/2017-11/konicki-coverage-improvement.pdf
2014 Task Force Report
https://www.census.gov/library/working-papers/2014/demo/2014-undercount-children.html
Two Bill O'Hare working papers when he was a research fellow in 2014
https://www.census.gov/srd/papers/pdf/ssm2014-03.pdf
https://www.census.gov/srd/papers/pdf/ssm2014-02.pdf
16

Undercount of Young Children Qualitative Evaluation, Version 2.1

1999 working paper
https://www.census.gov/population/www/documentation/twps0039/twps0039.html
Blogs
2010 Director's Blog (Bob Groves)
https://www.census.gov/newsroom/blogs/director/2010/03/children-count-too.html
2014 Research Matters Blog (Frank Vitrano)
https://www.census.gov/newsroom/blogs/research-matters/2014/08/counting-young-children-incensuses-and-surveys.html
2018 Director’s Blog (Ron Jarmin)
https://www.census.gov/newsroom/blogs/director/2018/07/improving_our_count.html

17

Undercount of Young Children Qualitative Evaluation, Version 2.1

Appendix B: Draft Focus Group Moderator Guide
A. INTRODUCTION (10 min.)
• Good evening, and thank you all for taking the time to come out and share your thoughts and
opinions. First a couple house-keeping items:
o Please silence your cell phones
o We have 90 minutes together.
o [Snacks/beverages?]
o Restrooms are [say where]; feel free to take a break anytime during our session
• Now I want to mention a few things about our discussion:
o As the moderator my job is to listen to you and encourage conversation; I have no stake
in the opinions you share, and no judgments. So please speak freely.
o There are no right or wrong answers; it is your thoughts, opinions and perceptions that
matter in here.
o Your participation is voluntary, and you don’t have to answer any question you don’t
want to.
o That said, you are here because your opinions are very important, and I do hope to hear
from everyone at some point this evening.
o Also, the goal of our conversation is not necessarily to agree with one another or come
to a consensus; it’s just to hear from each of you about your personal opinions and
experiences. So if you disagree with something that is shared, or agree, or somewhere
in-between, I’d really like to get your perspective.
o Everything you share will be kept among the research team, and you will not be
personally identified in any reports we prepare based on our conversation. We also ask
that we all respect the privacy of everyone in the room, and that you don’t share what
is discussed with others.
o To help me remember what’s been said, and not miss anything, we will be audio
recording and transcribing this session. This will help me focus on the conversation
without being distracted by taking notes. Any information that could identify you
personally will be removed from the transcripts, and the audio files and transcripts will
be destroyed when the report is complete.
• [If applicable] You may have noticed the glass behind me. Some of my team members are here
observing me so that we can all learn from the session and coach each other. [Introduce
observers to the group].
• I think that’s just about everything in terms of an introduction.
• Any questions?
Okay, let’s get started by introducing ourselves to each other. Please tell us: (1) your first name,
(2) how long you have lived in (name the city or town), and (3) a couple of your favorite TV
shows. I’ll go first. [Moderator introduce self; then invite others to do the same].
Great, very glad to meet all of you. Let’s start our discussion.
[TURN ON RECORDER]

18

Undercount of Young Children Qualitative Evaluation, Version 2.1

B. Past Experience with the Decennial Census (5 minutes)
• The topic of our discussion is the 2020 Census.
• Let me start by asking: does anyone here recall the census coming to your household in the
past – either a form in the mail, a census-taker at your doorstep or anything like that? [If nothing
probe: the last time would have been around March or April of 2010 – anyone remember? How
about 2000?]
• [Conduct open probing on the experience:
o Do you remember – was it a paper form, a census-taker at the door, did someone call
in to the census, etc?
o How did you became of aware of the census form/census taker? [Do you remember the
census form sitting around the kitchen table, and interviewer at the door, etc?]
o Do you recall how it was decided who would complete the census for everyone in the
household? Or was there no discussion and someone just did it?
o [other topics?]
• Ok great. Thanks for sharing that. We’ll get into lots of details and follow up on those
experiences later but this gives us a little real experience to get us focused on the discussion.
C. Within-household Negotiation Over Who Fills Out Form (10 minutes)
• So now let’s look to the future. If the census was happening now, and the form was mailed to
your household, would you or someone else be the person who would fill this form out?
• How do you think it would get decided – the person who would do the census for the
household?
• [If not self] How is that person related to you? And how is that person related to others in the
household?
• How would you rate that person’s knowledge of all the other household members?
D. EXERCISE: Filling Out the Census Form/Questionnaire (25 minutes)
NOTE: Depending on the timing of the focus groups, mock up the “Census Day” date on the form
to mimic the rough time frame of the real census. For example, if the focus groups are being held
in June, 2019, mock up the forms to say “June 1, 2019” instead of “April 1, 2020.”
• Next we’ll do an exercise. I’d like you to pretend that you received the Census 2020 form in
the mail and you are the one to fill it out on behalf of your household.
• Please open your folder and take out the form that looks like this [hold up Census 2020 form].
Take about 10 minutes to fill out the form as if you were doing it at home. Please try to make
it as realistic as you can, and don’t overthink it too much.
• When you’re finished we’ll talk through how you answered and why, and whether any
questions came up for you as you were filling it out.
• And please: DON’T WORRY! There are no right or wrong answers. We just want you to go
through the form in a way that is as realistic as possible. If there are any questions that you’re
not sure about, please just make a note to yourself on the form itself and we’ll talk about that
as a group.

19

Undercount of Young Children Qualitative Evaluation, Version 2.1

PERSON 1
Q1: Let’s start with Question #1, on the first page, at the bottom left. Everyone see that?

•

[Open probing; go around the table and try to draw out each person to contribute. Try to avoid
getting in to specific individuals (that will come later) and focus on general concepts]
o What number did they come up with and why
o What “rules” or “criteria” did they use to decide if someone should be listed or not
o Reasons they may have been unsure whether to list someone

Q2: Next, on the right-hand panel on top, is the question about any “additional” people:

•
•
•

Was anyone “staying here” on [date]?
[If yes: probe on whether they thought they should list that person on the census; why or why
not]
Did anyone check any of these boxes? Talk about why or why not.

20

Undercount of Young Children Qualitative Evaluation, Version 2.1

Q5: On the next page are questions about “Person 1”

•
•
•
•

•

Who did you all write down for “Person 1”?
Why did you choose that person?
Let’s take a look at the whole question. Just above the space for First/Last Name the question
asks “What is Person 1’s name?” Above that is a longer set of instructions starting with “Please
provide information…” Did anyone read the part as you were filling it out?
Let’s talk about that phrase “someone living here who pays the rent or owns this residence”:
o Can someone tell me in their own words what that means?
o Is that straightforward for most of you or is there some uncertainty?
o What if more than one person rents/owns the residence – how would you decide who
to write down as “Person 1”?
The next set of questions (6-10) ask about the sex, age, date of birth, race/ethnicity and
citizenship status of Person 1. Did anyone have trouble with any of those questions?

21

Undercount of Young Children Qualitative Evaluation, Version 2.1

PERSON 2
Q1: At the bottom right of Page 2 is an instruction about Person 2.
o Did anyone notice this?
o Did everyone turn to Page 3?
o Did anyone have doubts about where to list the next person?

•
•
•

How did you all decide who to list as Person 2?
What is the relationship between Person 1 and Person 2?
How knowledgeable is Person 1 about Person 2?

Q2: Next is a question asking whether the person lives/stays somewhere else.
• Did anyone check “no”? [discuss why]
• Anyone check “yes”? [discuss why]

22

Undercount of Young Children Qualitative Evaluation, Version 2.1

Q3: Asks how Person 2 is related to Person 1
• How did you all decide what to check here?
• What kinds of questions came up for you?
• Any difficulties, doubts or uncertainty?

PERSONS 3 thru End
[Repeat Person 2 series for remaining household members that participants listed on the form].
• Now let’s just talk about who got listed and who did not.
o Is there anyone you did not list that you think possibly should have been listed? [probe
who; why]
o What about people who may come and go through the household?
o How about babies and young children of parents who may sometimes stay other places,
not just this household?
o [If anyone mentions young children probe as much as possible on why they did or did
not include them; invite group discussion on the situation and solicit other opinions]
• Talk about how closely (or not closely) related you are to other household members.
o Is everyone related?
o If so, is it immediate family or more distant relatives?
o If there are non-relatives, are you friends or is it more of a roomer/boarder situation?
o Is there a mix of relatives and non-relatives?
• Do you think the names of people you listed on the form in the exercise would be different if
someone else in the household was filling out the form? [probe on why]
• What would you do if there were more than 10 people?

23

Undercount of Young Children Qualitative Evaluation, Version 2.1

E. Census Knowledge (10 minutes)
• Now that we’ve talked about everyone’s actual household situation, let’s talk about your
general perceptions about the census.
• How many Census forms do you think will be mailed to each household – one for each house
or apartment; one for each family; one for each adult; one for each person, including children?
• Do you think you are supposed to list everyone, regardless of age, on the census form, or only
adults? If you have any uncertainty at all please tell me more about that – what is your
perception of who to list on the form and why.
• Let’s talk more about children. Do any questions or issues come up for you, such as…
o Do you wonder if a child needs to be a certain age to be included?
o What about babies?
o Any reason to worry about others in household learning that you listed (or did not list)
a young child?
o Do you think you would list a child but another household member would not – or other
way around? Why would there be differences of opinion among household members
about whether to list a child?
o If you were uncertain about whether to list a child, what would you do? Would you…
 check Census website/mateirals?
 ask someone for advice? If so who? (another household member; relatives;
neighbors; friends)
o What do you think people in your community think about including young children on
the Census form?
o Do you have any reason to hesitate or fear listing young children? If so, why?
o What would you need to know to feel comfortable including a young child on the form?
F. Vignettes (10 minutes)
• Can you think of any neighbors or friends whose household is particularly complicated in terms
of who “lives” there?
• [If necessary] Present respondents with vignettes of known-to-be-problematic household
scenarios/compositions and ask how they think the form should be filled out.
G. Messaging (20 minutes)
1. When it comes to getting news about what’s going on around the world, in the country and in
your community, where do you turn?
a. Radio (ethnic?)
b. TV (ethnic?)
c. newspapers (ethnic?)
d. Facebook
e. Twitter
f. Instagram
g. Other social media/specify
2. What about individuals?
a. Family
b. Friends
24

Undercount of Young Children Qualitative Evaluation, Version 2.1

3.

4.
5.

6.

c. Neighbors
d. Doctors
e. Teachers
What about institutions?
a. Schools
b. Churches
c. Community centers
d. Social service agencies
e. Child care centers
Which of those do you trust? Which do you trust the most?
For the 2020 Census, there will be many options for how to complete it – including a paper
form, internet, census-taker at the door and calling in to a help line. Do you have a preference
for how to complete the census:
a. Online on a PC
b. Online on a mobile device
c. Paper
d. Calling in to Census
e. Other/specify
Do you access the internet from home?
a. If yes is it on a desktop or on a mobile device (phone, tablet etc.)
i. If only a phone, is it YOUR phone or the phone of someone else in the
household?
ii. Is the person whose phone it is likely to fill out the Census form for the
household?
iii. Do you/the person with the phone have data limits that would make you/them
reluctant to fill out the Census online?
b. If no:
i. Can you get internet access, e.g., at a library, community center,
friend/neighbor, internet café, etc.?
ii. Have you ever gone to one of those places to access the internet?
iii. Would you be very likely to go one of those places just to fill out the Census?

25

Undercount of Young Children Qualitative Evaluation, Version 2.1

Appendix C: DRAFT Cognitive Testing Protocol
A. PRE-INTERVIEW ACTIVIES
1. Introduction
Hello, my name is ______ and I work for the Census Bureau. Thank you very much for helping
us out today. [Make small talk; ask about any trouble finding the interview location or, if in their
home, make some compliments and thank them for allowing you to come to their home].
[If applicable: Before we get started let me mention that there are some observers watching the
interview in another room. They are only watching to get an idea of how the questions in the survey
we are going to talk about are working.]
Let me start by telling you a little bit about what we're doing here. Every 10 years the Census
Bureau conducts a head count and it is extremely important that this be as accurate as possible. So
we turn to people like you to find out if our questions make sense and are easy to understand and
answer.
2. Task
I'll ask you to go through the Census form and fill it out as if it was the real thing, in 2020. But I'm
mainly interested in your interpretations and reactions to the questions, and how you decided on
your answers. So after you’re done filling out the form, we’ll go through it one section at a time.
There are no right or wrong answers, we just want to know your thoughts and reactions to the
questions to help us make them better so that we can learn how to make the questions work for
everyone.
3. Confidentiality
The things we will talk about in the interview today will only be used in our research to help us
improve the survey. Your name will not be attached to anything you say. Direct quotes may be
used in research papers and professional presentations, but your name and any names you might
mention today will never be used in our reports and presentations. Your participation in this study
is completely voluntary. You can refuse to answer or skip over any particular questions. If at any
time you decide you do not want to go on, that is your choice and you may stop. Please feel free
to ask me any questions at any time.

26

Undercount of Young Children Qualitative Evaluation, Version 2.1

4. Audio Recording
I'd like to ask for your permission to audio record the session today. The main reason we record
these interviews is so that we don't have to rely on notes or our memories later. This allows me to
concentrate on what you're saying during the interview. What you say on the recording may be
used only for research purposes and cannot be told to anybody else not working on this project
except as required by law (20 U.S.C., § 9573). Do you have any questions about the recording?
• AFTER ANSWERING QUESTIONS AND GIVING FURTHER EXPLANATION,
CONTINUE.
• IF THE PARTICIPANT DOES NOT WANT TO BE RECORDED BUT IS WILLING TO
DO THE INTERVIEW, CONTINUE
• IS NO LONGER INETERESTED IN PARTICIPATING, THANK THE PARTICIPANT FOR
HIS/HER TIME AND END THE INTERVIEW.
5. Written Consent Form
Before we start, I’m going to give you a written consent form.
HAND THE INFORMED CONSENT DOCUMENT TO THE PARTICIPANT
This document explains the purpose of the study and how we will protect your information.
[After respondent signs]: Thank you for signing the consent form. I will also sign this form to
show that I agree to everything in the form.
•
•
•

ASK PARTICIPANT IF (S)HE HAS ANY QUESTIONS ABOUT THE CONSENT FORM
IF PARTICIPANT AGREES, OBTAIN THE RESPONDENT’S SIGNATURE;
INTERVIEWER SHOULD ALSO SIGN THE FORM
TAKE A SECOND COPY AND BOTH PARTICIPANT AND INTERVIEWER SIGN;
GIVE ONE COPY TO THE PARTICIPANT TO KEEP

6. Verbal Consent on Recording
IF PARTICIPANT PROVIDES CONSENT TO HAVE THE SESSION AUDIO-TAPED:
I’m going to turn on the recorder now, and once it is on, I’m going to ask for your permission to
record today, and then we will get started.
TURN ON TAPE RECORDER, ASK FOR PERMISSION TO RECORD ON THE TAPE, AND
BEGIN THE INTERVIEW.
OK, let’s begin.

27

Undercount of Young Children Qualitative Evaluation, Version 2.1

7. Mode Assignment (Paper, Internet Self-Response or NRFU)
Did you participate in the 2010 Census?
o Yes  Do you remember if you…
(a) mailed back the form
(b) answered the questions over the phone or
(c) a Census-taker came to your house or
(d) something else?
 Use answers from main screener, Q16a, to identify ISR versus paper mode
o No  NRFU mode
o DK/Don’t remember  NRFU mode
Q16 from Main Screener: Do you have at least one year of Internet experience?
o Yes  16a
o No (skip to next question)  PAPER mode
16a. How often do you use the Internet either on a computer or on a mobile device like a
smartphone or tablet?...Several times a day, about once a day, a few times a week, a few
times a month, or a few times a year?
o Several times a day  ISR mode
o About once a day  ISR mode
o A few times a week  ISR mode
o A few times a month  PAPER mode
o A few times a year  PAPER mode
o Not at all [do not read to respondent]  PAPER mode

28

Undercount of Young Children Qualitative Evaluation, Version 2.1

PAPER MODE
B. MAIN INTERVIEW
1. Let’s start with the number of people you wrote down in Question 1; you recorded [X].
• How did you come up with that number?
• Did you have any questions about who to include? [why/why not]
• Did you have questions about who to exclude? [why/why not]
• At the beginning, the set of instructions asks you to count “people living in this
house…” Can you tell me how you interpreted the word “living”?
• The set of instructions and Question 1 are worded like this (read below). How do you
interpret each of those phrases? Do they mean the same thing to you or something
different?
o Instructions: “count the people living in this house.”
o Question 1: “How many people were living or staying in this house…”
• [Unless the respondent talks about the instructions, hold off discussing those until
you’ve gone through all the names of individuals listed on the form]

29

Undercount of Young Children Qualitative Evaluation, Version 2.1

30

Undercount of Young Children Qualitative Evaluation, Version 2.1

2. Question 2 asks about “additional” people.
• What was your first reaction to this question?
• How did you decide to check the box(se) that you checked?
• [Probe on each response category checked]
i. Meaning of terms/category
ii. How they decided it applied to their household
iii. Why they did not initially include that/those person/people in Q1
• [Probe on meaning of terms/category of each response category not checked]
• Let’s talk about the phrase “staying here on April 1, 2020”
i. Can you tell me, in your own words, what that phrase means? What are some
examples of what you think we mean by someone “staying here on April 1,
2020”?
ii. Now let’s talk about the date.
1. April 1 is what we call “Census Day.” We need to count everyone
once and only once, in the right place. To do that, we pick a day so we
can get a snapshot of the population at a particular point in time.
2. Next year when we do the Census for real, we’ll be out in the field
collecting data for about a 2-month window, before and after April 1.
3. Now that I’ve explained that, would you answer Q1 or Q2 any
differently? [probe on how their notion of “staying here on April 1”
relates to their answers to Q1 and Q2].
• Did you notice you could check more than one box?

31

Undercount of Young Children Qualitative Evaluation, Version 2.1

3. Now let’s talk about each person you listed on the form. First was [yourself/NAME].
• Why did you list that person as “Person 1”?
• Did you have any doubts about that?
• [Open probing on who Person 1 is, if it’s not the respondent]
o Is the person a relative, landlord, etc?
o Why does the respondent consider Person 1 “living or staying” at the
residence?
• Was there another person you considered listing as Person 1?
• Let’s talk about the instructions in Q5:
o Phrase “someone living here who pays the rent or owns this residence”?
o How did you decide if someone who owns/rents “lives” at the “residence”?
o If you decided no one who lives there own/rents, how did you decide who to
select as “Person 1”?

32

Undercount of Young Children Qualitative Evaluation, Version 2.1

4. How about Person 2?
• How did you decide who to list here?
• [open probing on who this person is to the respondent and why the respondent
considered him/her to be “living” at the residence]

•

How did you decide on your answer for Q2?

•

What about the phrase “usually live or stay” with regard to [Person 2] – did you have
any questions about what that meant?
[If Person 2 sometimes stays somewhere else] Where does [Person 2] sometimes
stay?
[go thru response categories one at a time; ask respondent to say what these places
mean to them]

•
•

33

Undercount of Young Children Qualitative Evaluation, Version 2.1

•
•

How did you decide what to answer for this question?
[If respondent is NOT Person 1]: Do you think [Person 1] would have answered this
question the same way you did? That is, you selected [category X] for how Person 2 is
related to [Person 1]. How certain do you feel that if [Person 1] was filling out the form,
they would also select [category X]?

34

Undercount of Young Children Qualitative Evaluation, Version 2.1

5. Now let’s talk about Person 3. [Repeat protocol for Person 2]

6. Now let’s talk about Person 4. [Repeat protocol for Person 2]

7. Now let’s talk about Person 5. [Repeat protocol for Person 2]

35

Undercount of Young Children Qualitative Evaluation, Version 2.1

8. After you finished answering questions for each person 1-6 about (sex, age, race), there was
an instruction at the bottom right (shown below).
• Did you notice this instruction?
• Did you remember how many people you had listed in Question 1, or did you have to go
back and look?

9. Now let’s talk about Person 6. [Repeat protocol for Person 2]

36

Undercount of Young Children Qualitative Evaluation, Version 2.1

10. Now let’s talk about the instructions for what to do if the number of people listed in Q1 was
7 or more. These instructions were on the bottom right corner of the page about Person 6.
• Did you notice the instruction?
• Did you go back and look at what you wrote down in Q1?
• When you first wrote your answer to Q1, did you feel fairly certain or did you have
some doubts?
• As you were writing down Persons 2-6, did you change your mind about who you
should have included in the count you recorded in Q1? [Probe for whether they
thought they should have included anyone they initially left out of the count, or
whether they counted someone they later thought should not have been included].

37

Undercount of Young Children Qualitative Evaluation, Version 2.1

•
•
•
•
•
•

If your answer to Q1 was 7 or more, did you know where to record them?
For Person 7 how did you decide what to answer for “Related to Person 1?”
And how about for Person 8? (How did you decide what to answer for “Related to
Person 1?”)
And for Person 9?
And Person 10?
[If Q1 was more than 10 people]: What did you think when you got to Person 10 on
the form?

38

Undercount of Young Children Qualitative Evaluation, Version 2.1

39

Undercount of Young Children Qualitative Evaluation, Version 2.1

11. Now let’s go back to the very beginning and talk about the instructions.
• Are there any babies or young children who sometimes stay at the residence?
• [Probe on respondents’ understanding/definition of each phrase]
o “Count the people living in this…”
o “live and sleep here most of the time”
o “without a permanent place to live”
o Let’s look at the that second half of the instructions –
 institutions
 college
 Armed Forces
 nursing home
 jail
 prison
 detention facility
Did this list make you think of any other institutions? [probe for which
ones; why].

40

Undercount of Young Children Qualitative Evaluation, Version 2.1

12. Vignettes: Now let’s talk about other situations – not your own household. Let’s go back to
the flyer you may have seen advertising the study, where we said:
You may be eligible if you live …
• with at least 4 people OR
• with 3 or more generations (e.g., grandparents, parents, grandchildren) OR
• with distant relatives, such as nephews/nieces, cousins, great grandchildren, etc. OR
• where not everyone is related to each other OR
• where someone has moved in or out recently
•
•

•

Can you think of any household that meets at least one of these descriptions – any
neighbors, friends, family, etc?
[if so] I’m going to give you another blank Census form and ask you to pretend you are
filling it out for that household. First try to think of who within that household would be
the one to fill out the form for everyone. Then try to put yourself in their shoes and go
through the form, ok?
[On blank form, highlight the following questions and ask them to answer these (and skip
the rest)]:
o Q1, Q2, Q5
o Person2-6: Q1-Q3
o Persons 7-10 (all Qs)
o What would they do if there were more than 10 people?

NOTE: If there’s still time in the interview, we could administer some hypothetical vignettes.
Based on the literature (Schwede et al 2018 AAPOR poster) the “top 3” household profiles
where most young children were initially left off the form are:
• households with nonrelatives (26%)
• multi-generation households (21%)
• family households with other relatives (15%)
Based on O’Hare and Pollard presentation (Southern Demographic Association October
2018), the top independent variables for households with omitted young children:
• Young children living with nonrelatives
• Large households (7+ people)
• Linguistically isolated
• Child living with Grandparent
• Mobility
• Percent minority families
The scenarios below capture some of these characteristics. We could tweak, invent others,
etc.
A. Woman in her 40s lives with her two teenage children, and her mother. They have an extra
room and they are renting it out to someone they found on Craigslist, and the renter lives
there with her baby.
41

Undercount of Young Children Qualitative Evaluation, Version 2.1

B. Married couple in their 60s live with their daughter (in her 20s). Daughter’s friend, also in
her 20s, has a toddler. Friend and toddler have been staying there for the past few months
and will stay there until they can find a place of their own that they can afford.
C. Man in his 60s lives with his elderly parents, and with his son (in his 30s) and his son’s
girlfriend. The girlfriend has a toddler, and on weekends the toddler stays with her
biological father, who lives a few blocks away.
D. Married couple in their 30s live with their two young daughters, and their elderly father.
The husband’s cousin, who just graduated from college, moved out of her college housing
and is staying with them with her baby rent-free while she completes a 6-month internship.
The cousin doesn’t know where she and her baby will live when the internship is over.

42

Undercount of Young Children Qualitative Evaluation, Version 2.1

13. Finally, these are the questions we ask about each person listed on the form.
[not sure how to go about these probes – ask at general level, not specific to any one hh
member, or pick a person, or what. It’ll be too long/too much to go thru each hh member for
each question…thoughts?]

43

Undercount of Young Children Qualitative Evaluation, Version 2.1

44