2010 ACS Content Test Evaluation Report - Public Assistance

January 26, 2012

2012 AMERICAN COMMUNITY SURVEY RESEARCH AND EVALUATION REPORT
MEMORANDUM SERIES #ACS12-RER-09

MEMORANDUM FOR

ACS Research and Evaluation Steering Committee

From:

Charles Nelson /Signed/
Chief
Social, Economic and Housing Statistics Division

Prepared by:

Edward J. Welniak Jr., Amanda R. Noss and Kirby G. Posey
Income Statistics Branch
Social, Economic, and Housing Statistics Division

Subject:

2010 ACS Content Test Evaluation Report Covering Public
Assistance

Attached is the final 2010 ACS Content Test Evaluation Report Covering Public Assistance.
This report describes the results of the 2010 Content Test with a change in wording to the Public
Assistance question. Final results indicate the test question wording will not be implemented for
2013 ACS. If you have any questions about this report, please contact Edward Welniak at
(301)763-5533 or Amanda Noss at (301)763-6675.
Attachment: (2010 ACS Content Test Evaluation Report Covering Public Assistance)

cc:
ACS Research and Evaluation Steering Committee
ACS Research and Evaluation Team
Todd Hughes
(ACSO)
Debra Klein
Donna Daily
Jennifer Childs
Jennifer Tancreto
(DSSD)
Tony Tersine
Mary Davis
Charles Nelson
(SEHSD)

American Community Survey Research and Evaluation Program
January 26, 2012

2010 ACS Content Test
Evaluation Report Covering
Public Assistance
FINAL REPORT

Edward J. Welniak Jr.
Amanda R. Noss
Kirby G. Posey
Social, Economic,
and
Housing Statistics Division

Intentionally Blank

TABLE OF CONTENTS
EXECUTIVE SUMMARY ........................................................................................................... iv
1. BACKGROUND ........................................................................................................................1
1.1 Motivation for the 2010 Content Test ..................................................................................1
1.2 Previous Testing or Analysis ...............................................................................................2
1.3 Recommendations from Cognitive Testing .........................................................................2
1.4 Recommendations from the Expert Review Panel ..............................................................3
2. SELECTION CRITERIA ...........................................................................................................3
3. METHODOLOGY .....................................................................................................................4
3.1 Data Collection Methods .....................................................................................................4
3.2 Sample Design .....................................................................................................................5
3.3 Methodology Specific to Cash Public Assistance ...............................................................5
4. LIMITATIONS ...........................................................................................................................6
5. RESEARCH QUESTIONS AND RESULTS ............................................................................6
5.1 Response to the Content Test and Content Follow-Up .......................................................6
5.2 Is the response distribution of cash public assistance income comparable to the
Current Population Survey’s Annual Social and Economic Supplement (CPS ASEC)
distribution of cash public assistance income? ..........................................................................7
5.3 Do the changes to the cash public assistance question raise the estimate of persons
receiving cash public assistance? ...............................................................................................7
5.4 Do the changes to the cash public assistance question raise the estimate of cash
public assistance income? ..........................................................................................................8
5.5 Do the changes to the cash public assistance question lower the item missing data
rates ? .........................................................................................................................................9
5.6 Do the changes to the cash public assistance question lower response error (i.e.,
bias) in the estimates of cash public assistance recipiency and cash public assistance
income? ......................................................................................................................................9
5.7 For each mode of data collection, do the changes to the cash public assistance
question affect the item missing data rates, the estimates of recipiency and cash public
assistance income, or the response error (i.e., bias)? ...............................................................11

i

5.8 For each mail response stratum, do the changes to the cash public assistance
question affect the item missing data rates, the estimates of recipiency and cash public
assistance income, or response error (i.e., bias)? ...................................................................12
5.9 Does either question version elicit respondent or interviewer behaviors that may
contribute to interviewer or respondent error?.........................................................................14
5.10 For the Hispanic and Black population subgroups, do the changes to the cash
public assistance question affect the estimate of recipiency, item missing data rate, or
reliability of the data? ..............................................................................................................14
6. SUMMARY ..............................................................................................................................15
References ......................................................................................................................................16
Acknowledgements ........................................................................................................................16
Appendix A: Tables .................................................................................................................... A-1
Appendix B: Images of the Mail Versions of the Control and Test Questions/Cognitive
Testing wording ...........................................................................................................................B-1
Appendix C: CATI and CAPI Versions of the Control and Test Questions ...............................C-1
Appendix D: Flow of the Content Follow-Up Interview ............................................................ D-1
Appendix E: Information Page .................................................................................................... E-1
Appendix F: CFU Wording ......................................................................................................... F-1

LIST OF TABLES
Table 1. Content Test Response Rate Comparison Between the Control and Test
Treatments........................................................................................................................................7
Table 2. Response Distribution CPS/ASEC ....................................................................................7
Table 3. Recipiency Rates, Control versus Test ..............................................................................8
Table 4. Mean and Median Estimates of Cash Public Assistance Income ......................................9
Table 5. Item Missing Data Rates, Control versus Test ..................................................................9
Table 6. Net Difference Rates, Control versus Test ......................................................................10
Table 7. Recipiency Net Difference Rates by mode of Interview .................................................11
Table 8a. Net Difference Rates-High Response Stratum...............................................................13
Table 8b. Net Difference Rates-Low Response Stratum ...............................................................13
Table 8c. Means and Medians-Low Response Stratum .................................................................14
Table 10a. Recipiency-Cash Public Assistance Income by Hispanic Origin and Race.................14
Table 10b. Median Estimates of Cash Public Assistance Income by Hispanic Origin .................15
ii

Table 10c. Item Missing Data Rates for Cash Public Assistance Income Recipiency and
Amount by Hispanic Origin and Race ...........................................................................................15
Table 10d. Net Difference Rates for Cash Public Assistance Income Recipiency by
Hispanic Origin and Race ..............................................................................................................15

Appendix Tables
Table A-1. Recipiency ………………… ................................................................................... A-1
Table A-2. Item Missing Data Rates for Recipiency .................................................................. A-1
Table A-3. Net Difference Rates-CATI/CAPI............................................................................ A-1
Table A-4. Net Difference Rates-CATI ...................................................................................... A-2
Table A-5. Net Difference Rates-CAPI ...................................................................................... A-2
Table A-6. High Response Stratum-Recipiency ......................................................................... A-3
Table A-7. Item Missing Data Rates- High Response Stratum .................................................. A-3
Table A-8. Mean and Medians- High Response Stratum ........................................................... A-3
Table A-9. Low Response Stratum-Recipiency.......................................................................... A-3
Table A-10. Item Missing Data Rates-Low Response Stratum .................................................. A-4

iii

EXECUTIVE SUMMARY
Test Objective
In late August through mid-December 2010, the Census Bureau conducted a field test of
new and revised content in the 2010 American Community Survey (ACS) Content Test.
The results of that testing will help determine the content to be incorporated into the ACS
in 2013.
Research shows that the receipt of cash public assistance (PA) or welfare in the ACS is
much lower than administrative counts (see Lynch et. al., 2008). Program eligibility can
vary by household structure and depends on the presence of children and householder.
Participation can also be sporadic throughout the year. Researchers believe that the ACS
question does not make it clear to the respondent that they should include participation on
behalf of children or that they should report even single month participation in a cash PA
program.
Methodology
The Content Test compared two versions of the cash public assistance income question.
The control version replicated the wording and response categories used in the current
production ACS question (CATI/CAPI wording shown), which are as follows:
-“Did you receive any cash public assistance or welfare payments from the state or local
welfare office DURING THE PAST 12 MONTHS?”
<1> Yes
<2> No
-

If yes:
“What was the amount?”

The test version for CATI/CAPI included the following changes to the control version of
the cash public assistance question:
-“Did you receive any welfare payments or cash assistance from the state or local welfare
office, for yourself or any children in this household DURING THE PAST 12
MONTHS?” Include all assistance, even if for only one month. Do NOT include benefits
from food, energy, or rental assistance programs.
<1> Yes
<2> No
-

If yes:
“What was the amount?”

iv

The test question emphasizes reporting amounts received for a child as well as not
including benefits for food, energy or rental assistance programs.

Research Questions and Results
Is the response distribution of cash public assistance income comparable to the
Current Population Survey’s Annual Social and Economic Supplement (CPS
ASEC) distribution of cash public assistance income?
Yes. The overall distribution of cash public assistance income for the test version is
comparable to that of the CPS ASEC. However, formal comparisons were not made
since the Content Test data were not edited or imputed, adjusted for nonresponse, nor
raked to known population totals.
Do the changes to the cash public assistance question raise the estimate of persons
receiving cash public assistance?
No. The changes to cash public assistance do not significantly raise the estimate of
persons receiving cash public assistance income.
Do the changes to the cash public assistance question raise the estimate of cash
public assistance income?
No. The mean and median estimates of cash public assistance income are not
significantly higher in the test version of the question compared to the control.
Do the changes to the cash public assistance question lower the item missing data
rates?
No. The changes to the question do not significantly lower the item missing data rates.
Instead, the item missing data rates for “cash public assistance income” are significantly
higher for the test version compared to the control.
Do the changes to the cash public assistance question lower response error (i.e., bias)
in the estimates of cash public assistance recipiency and cash public assistance
income?
Yes. But the changes to the cash public assistance question significantly reduced the
overestimate of recipiency which is the opposite of what the test version was aiming to
accomplish, therefore producing negative results. No significant changes were found for
the income amounts.
For each mode of data collection, do the changes to the cash public assistance
question affect the item missing data rates, the estimates of recipiency and cash
public assistance income, or the response error (i.e., bias)?
v

Yes. The test version reduced the overestimate of public assistance recipiency for mail
responses compared to the control. All other measures by mode of data collection
showed no statistical differences.
For each mail response stratum, do the changes to the cash public assistance
question affect the item missing data rates, the estimates of recipiency and cash
public assistance income, or response error (i.e., bias)?
Yes. Response error for cash public assistance income recipiency for both mail response
strata was significantly lower in the test than control. The median estimate for cash
public assistance income was significantly higher for the test version than the control in
the low response stratum. All other measures by stratum showed no statistical
differences.
Does either question version elicit respondent or interviewer behaviors that may
contribute to interviewer or respondent error?
Results indicate that interviewer behavior on the test questions was not as good as
interviewer behavior for the control questions. Review of the behavior coder notes
indicate that interviewers stopped reading the test question at “… during the past 12
months” and often dropped the last sentence (“Do not include …”). For respondent
behavior, the test series performed better than the control.
For the Hispanic and Black population subgroups, do the changes to the cash public
assistance question affect the estimate of recipiency, item missing data rate, or
reliability of the data?
The results are mixed. The recipiency rate for the test version was statistically higher
than the control for Hispanics. The test version produced a significantly lower item
missing data rate than the control version of the Hispanics as well. However, the test
version also resulted in an increase in the overestimate of recipiency among Hispanics.
The median estimate of cash public assistance income was statistically higher in the test
version of the question among Blacks. The test version also reduced the overestimate of
recipiency amount among Blacks.
Recommendation
Health and Human Services (HHS), the sponsor for the change to the Cash Public
Assistance question, suggested not to proceed with this change for 2013. There were
several positive and negative results, however there were more negative results. The
results are further discussed below. The goal was to capture more households with public
assistance income and this was not achieved with the test version.

vi

1. BACKGROUND
1.1 Motivation for the 2010 ACS Content Test
To evaluate proposed changes to the content of the American Community Survey (ACS),
the Census Bureau conducted the 2010 ACS Content Test. The objective of the ACS
Content Test, for both new and existing questions, was to determine the impact of
changing question wording, response categories, and redefinition of underlying
constructs on the quality of data collected.
Through the Office of Management and Budget (OMB) Interagency Committee on the
ACS, subject matter experts from the Census Bureau and key data users from other
federal agencies collaborated in identifying revised and new questions for inclusion in the
Content Test. The suggested new and revised questions affected both the housing and
detailed person sections of the ACS questionnaire.
In the housing section, the food stamps question was altered to reflect a name change for
the food stamps program. In addition, a series of new questions were added related to
household computer ownership and Internet subscription.
Several changes were made in the detailed person section. First, a change in data needs
for the veteran series led to a revised set of response categories for the veteran status and
period of military service questions. Second, the question wording of the cash public
assistance income question was modified to address under-reporting of assistance on
behalf of children and single payment recipients. Third, to simplify the income questions
related to wages (wages, salary, commissions, bonuses, or tips) and property income
(interest, dividends, rental income, royalty income or income from estates and trusts),
these questions were broken up into smaller questions for the Computer-Assisted
Telephone Interviewing (CATI) and Computer-Assisted Personal Interviewing (CAPI)
instruments only. Fourth, a set of new questions on parental place of birth were added to
allow data users to divide the population into “first generation” (the foreign born),
“second generation” (the children of immigrants), and “third or higher generation”
(native born with no foreign-born parents).
To meet the test objective of the 2010 ACS Content Test, analysts evaluated changes to
question wording, response categories, instructions, and examples relative to a control
version of the question or another version for new questions. Specifically, this report
discusses the cash public assistance income question.

1

1.2 Previous Testing or Analysis
Research shows that the receipt of Cash Public Assistance (PA) or welfare in the ACS is
much lower than administrative counts (see Lynch, Resnick, 2008). Participation can be
sporadic throughout the year. Researchers at Health and Human Services (HHS) believe
that the ACS question does not make it clear to the respondent that they should include
participation on behalf of children or that they should report even single month
participation in a PA program. In many households only the children are eligible for
assistance. In many states children 15 years old and younger qualify for such assistance
programs. Households may be receiving income based on these children and individuals
completing the survey may not be counting this type of income when reporting in the
ACS.

1.3 Recommendations from Cognitive Testing
Prior to conducting the Content Test, the Research Triangle Institute (RTI), Westat, and
Research Support Services (RSS) conducted cognitive interviewing, under contract, to
assist in identifying a final set of questions for the field test. Multiple versions of each
question topic were tested with the goal of choosing the best one for the revised questions
and the best two for the new questions. The questions were pretested in the three modes
used in the ACS data collection (paper, telephone interview, and personal interview) in
English and Spanish. Cognitive interviews consisted of one-on-one interviews using the
proposed questions in the context of the ACS survey. Survey methodologists also
conducted respondent debriefings.
RTI tested two versions of proposed new ACS questions about cash public assistance as a
source of income. Both versions included phrasing to encourage respondents to report
public assistance as an income source even if they had received it only once during the
12-month reference period. One version did this by stating “even if for only one month;”
the other version stated “even if for only one payment.” A further variation concerned the
ordering of the two key phrases within the question: Version 1 stated “…even if for only
one month, for this person or any children in this household,” while Version 2 stated “…
for this person or any children in this household, even if only one payment.”
See Appendix B for wording.
The findings of this testing did not clearly point to one question version being better than
the other. The problems and difficulties observed were almost evenly distributed across
the two versions and unrelated to the wording variations of interest. The main
recommendation was to place more emphasis on the instructions not to include benefits
from other programs, and use the word “month” rather than “payment.” Therefore, our
recommendation was a modified version of the question: “DURING THE PAST 12
MONTHS, did [/yourself] receive any welfare payments or cash assistance from
the state or local welfare office for [/yourself] or any children in this household,
even if for only 1 month? Do NOT include benefits from any other type of assistance,
such as SSI, food, energy, or rental assistance programs.”

2

For more information and complete question wording, see RTI International (2009),
“Cognitive Testing of the American Community Survey Content Test Items.”

1.4 Recommendations from the Expert Review Panel
Following the cognitive testing, an expert review panel, composed of government survey
methodology experts, reviewed and added changes to the final question versions
proposed to move forward from the cognitive testing into the field test. The proposed
changes for each question topic were approved by the corresponding OMB interagency
subcommittee responsible for initiating the research. The OMB provided final approval
of the proposed changes.
The expert panel’s recommendation was to change the CATI/CAPI question wording
slightly to include the bolded text below. A similar change was made to the mail
question.
“DURING THE PAST 12 MONTHS, did you receive any welfare payments or cash
assistance from the state or local welfare office for yourself or any children in this
household, even if for only one month? Do NOT include benefits from food, energy,
or rental assistance programs.” Respondents were to answer “yes” or “no.” See
appendix B and C for final question wording.
The panel also recommended including an interviewer instruction for this question in the
CATI / CAPI instruments that lists the local welfare program name(s).

2. SELECTION CRITERIA
The research questions in sections 5.2 through 5.10 appear in order of importance for the
decision of whether the test version of the question is better than the control question.
The selection criteria below are also shown in order of importance to the decision.
The overall distribution of cash public assistance income for the test version should be
comparable to that of the CPS ASEC.
An increase in cash public assistance receipt and the amount of cash public assistance
received in the test version implies a positive change since this item is historically
underestimated.
The item missing data rates and response error (i.e., bias) will be considered together
when determining whether the test version performs better.

3

3. METHODOLOGY
3.1 Data Collection Methods
The initial stages of the Content Test consisted of content determination, cognitive
laboratory pretesting, and expert reviews for the purpose of developing alternate versions
of question content. The field test portion of the ACS Content Test used the data
collection methodology currently used in the production ACS (i.e., mail questionnaire,
follow-up CATI, and follow-up CAPI) with an added reinterview conducted via a CATI
instrument known as the Content Follow-Up (CFU). Additional data were collected on
respondent and interviewer behavior during the field test via Computer Audio Recorded
Interviewing (CARI) technologies for a subset of respondents during the CATI and CAPI
follow-up modes of data collection.
The Content Test followed the same schedule and procedures for the mail, CATI, and
CAPI operations as the September 2010 ACS production panel. Questionnaires were
mailed to sampled households at the end of August 2010. The Content Test used an
English-only mail form but the automated instruments (CATI, CAPI, and CFU) included
both English and Spanish versions. Households not responding by mail and for which we
had a phone number were contacted for a CATI interview during the month of October
2010. In November 2010, Census Bureau field representatives visited a sample of
households that did not respond by mail or CATI to attempt a CAPI interview. The CAPI
operations ended December 2, 2010.
The field test included a CATI CFU reinterview to collect additional measures for the
study of response error. This operation started approximately two weeks after the initial
mail out of questionnaires and ended two weeks after the end of the CAPI follow-up data
collection operation. The CFU included all occupied households for which we received a
response in the original interview and had a telephone number. A response was defined
as a case where the household provided data through at least the first person’s place of
birth question for mail cases or at least a sufficient partial interview for CATI/CAPI
interviews. The reinterview was conducted about 2 to 4 weeks after the original
interview and with the original respondent when possible. Note that the CFU CATI
interview was an abbreviated version of the original Content Test interview. The CFU
instrument included the basic demographic section and only those questions preceding
the questions being tested in the housing and the detailed person sections to provide
context (see Appendix D for the flow of the CFU instrument).
The ACS Content Test did not include all of the production data collection operations and
processes. First, while the Telephone Questionnaire Assistance (TQA) program’s tollfree number was available to Content Test respondents for assistance, the CATI
instrument did not include content changes from the Content Test. Therefore data
collected from Content Test respondents via TQA CATI interview were not included in
our analysis. Second, since our objective was to study response error using unedited
data, the Content Test excluded the Failed Edit Follow-up (FEFU) CATI operation and
the edit and imputation data processes.
4

3.2 Sample Design
The 2010 Content Test consisted of a national sample of 70,000 residential addresses in
the contiguous United States (the sample universe did not include Puerto Rico, Alaska,
and Hawaii). The sample design for the Content Test was largely based on the ACS
production sample design with some modifications to meet the test objectives. The
modifications included adding an additional level of stratification by stratifying addresses
into high and low mail response areas, over-sampling addresses from the low mail
response areas to ensure equal response from both strata, and sampling units as pairs.
The high and low mail response strata were defined based on ACS mail response rates at
the tract-level. The paired sample selection formed pairs by first systematically sampling
an address within the defined sampling strata and then pairing that address with the
address listed next in the geographically sorted list. However, the pair was not likely
comprised of neighboring addresses. One member of the pair was randomly assigned to
the control group and the other member was assigned to the test group. Those addresses
assigned to the test group received the revised ACS questions and the questions new to
the ACS. The control group received the current questions on the production ACS as
well as different versions of the new questions.
Another modification to the production ACS sample design included adding a third
sampling stage. At the first stage, the production 2010 ACS first stage sample was used
as the Content Test first stage sample. At the second stage, all housing units in the ACS
first stage sample not selected in the production 2010 ACS second-stage sample were
selected as the Content Test second-stage sample. In addition, any units that were
selected to be in other operations (e.g., training, other tests, etc.) were not selected in the
Content Test second stage sample. At the third stage, addresses were selected using a
sampling method similar to the production ACS second stage sample design with the
exception of adding the high and low mail response stratification.

3.3 Methodology Specific to the Cash Public Assistance Income
Only persons 15 or older were considered in the universe for the analysis, since all
income questions are only asked of this universe. On the mail questionnaire, public
assistance recipiency was determined if there was a Yes response in the recipiency field
or if a dollar amount greater than zero was in the amount field.
The ASEC questions were used to ask the question a second time and then make
inferences and use as a “true measure”. See Appendix F for CFU question wording.

5

4. LIMITATIONS
Control and test CATI/CAPI workload assignments were not assigned using an
interpenetrated experimental design. That is, interviewers were allowed to administer
interviews for both control and test cases, in addition to production ACS cases. The
potential risk of this approach is the introduction of a cross-contamination or carry-over
effect due to the interviewer administering multiple versions of the same question item.
Interviewers are trained to read the questions verbatim to minimize this risk, but there
still exists the possibility that an interviewer may deviate from the scripted wording of
one question version to another. This could potentially mask a treatment effect from the
data collected.
The CFU reinterview was not conducted in the same mode of data collection for
households that responded by mail or CAPI in the original interview since CFU
interviews were only administered using a CATI mode of data collection. As a result, the
data quality measures derived from the reinterview may include some bias due to the
differences in mode of data collection.
Respondents needed to provide a telephone number in the original Content Test interview
or the Census Bureau had to be able to find a telephone number for that unit through
reverse address look-up to be included in the CFU interview. As a result, 18.4 percent of
the responding households from the original interview were not eligible for the CFU
reinterview.
We did not have the same respondent in the CFU that we had in the original interview for
9.1 percent of the CFU cases. This means that differences between the original
interview and the CFU for these cases could be due in part to having different people
answering the questions.
The Content Test does not include the production weighting adjustments for seasonal
variations in ACS response patterns, nonresponse bias, and under-coverage bias. The
CFU portion of the Content Test did include a unit nonresponse adjustment for those
Content Test cases that responded to the Content Test, but failed to respond to the CFU.
As a result, the statistics derived from the Content Test data do not provide the same level
of inference as the production ACS to the entire population of housing units and persons
in the contiguous United States.

5. RESEARCH QUESTIONS AND RESULTS
5.1 Response to the Content Test and Content Follow-Up
Table 1 shows the unit response rates for each of the modes of data collection and all
modes combined (excluding CFU) by the control and test groups. The comparison
between control and test show that respondent participation was similar for both control
and test for each of the modes of data collection and all modes combined, with the
exception of the CATI mode. The test treatment produces a CATI rate of response that is
6

3 percentage points higher compared to that of the control. We are not able to explain the
increase in response due to the test treatment for the CATI mode of data collection other
than by random occurrence given that the conditions affecting unit response were
equivalent between the test and control groups.
Table 1. Content Test Response Rate Comparisons Between the Control and Test Treatments
Standard
Standard
Test Standard
Test
Error
Control
Error
Control
Error
Mode
(%)
(%)
(%)
(%)
(%)
(%)
Significant
All Modes
(CFU
95.4
0.2
95.7
0.2
-0.3
0.3
No
excluded)
Mail
58.1
0.5
57.7
0.5
0.5
0.7
No
CATI
52.6
1.2
49.6
1.0
3.0
1.5
Yes
CAPI
90.4
0.5
91.5
0.5
-1.1
0.7
No
CFU
54.3
0.5
53.5
0.6
0.8
0.7
No

5.2 Is the response distribution of cash public assistance income comparable to the
Current Population Survey’s Annual Social and Economic Supplement (CPS
ASEC) distribution of cash public assistance income?
Table 2 shows the response distributions of the test and control versions compared to the
2010 CPS ASEC. The overall distribution of cash public assistance income for the test
version is comparable to that of the 2010 CPS ASEC. Formal statistical comparisons
were not made since the Content Test data were not edited or imputed, nor were there
adjustments for non-response or raking to known population totals. The differences
between the CPS ASEC and ACS Test and control questions in the $1,000 to $4,999 and
the $5,000 to $9,999 income intervals can be partially explained by the clustering of
responses at rounding points ($5,000, $6,000, $7,000, $8,000, $9,000 and $10,000) in the
ACS. The clustering in the ASEC is not as pronounced, likely due to the fact that the
ASEC allows respondents to report income sub-annually (monthly) along with
periodicity (number of months received), which the ACS does not. Similar differences
exist between the ASEC and the ACS production distribution.
Table 2. Response Distribution
Category

ASEC
Estimate
(%)
0.5
3.0
8.5
12.7
56.6

Standard
Error
(%)
NA
NA
NA
NA
NA

Test
Estimate
(%)
(n=507)
1.0
4.5
9.3
13.0
45.8

Standard
Error (%)

Control
Estimate
(%)
(n=622)
0.0
5.4
9.6
14.8
46.0

Standard
Error (%)

$1 or $2
1.0
0.3
$3 to $199
1.3
1.0
$200 to $499
1.8
1.8
$500 to $999
2.6
2.3
$1,000 to
3.3
2.8
$4,999
$5,000 to
15.1
NA
20.2
3.3
18.7
2.3
$9,999
$10,000 or more
3.6
NA
6.1
1.4
5.5
1.2
Total:
100.0
100.0
100.0
Source: U.S. Census Bureau, 2010 Current Population Survey, Annual Social and Economic Supplement.

7

5.3 Do the changes to the cash public assistance question raise the estimate of
persons receiving cash public assistance?
To measure this, the proportion of persons receiving cash public assistance, or the
recipiency rates, were computed and compared between the control and test versions.
Table 3 shows public assistance recipiency rates for the control and test groups and the
difference between the test and control groups. A one-sided test was used to determine if
the test group had a statistically significant larger recipiency proportion using an =
0.10. The changes to cash public assistance did not significantly raise the estimate of
persons receiving cash public assistance income. The main goal of this content test was to
capture more household receiving this source of income. Table 3 shows that this goal was
not met with the test version.

Table 3. Recipiency Rates
Test
Standard
Control
Standard
Test –
Standard
Estimate
Error
Estimate
Error
Control
Error
(%)
(%)
(%)
(%)
(%)
(%)
Significance
Recipiency
1.4
0.1
1.6
0.1
-0.2
0.1
No
Rate
Source: U.S. Census Bureau, 2010 American Community Survey Content Test, September to December
2010

5.4 Do the changes to the cash public assistance question raise the estimate of cash
public assistance income?
To measure this, the mean and median estimates of PA amounts were computed and
compared between the control and test versions. PA recipients were included in the
computations even if the PA amount was 0.
Table 4 shows median and mean estimates of cash public assistance income for the test
and control groups and the difference between the test and control groups. A one-sided
test was used to determine if the test group had a statistically significant larger median
and mean using an α = 0.10. The results showed that the mean and median estimates of
cash public assistance income are not significantly higher in the test version of the
question. Had there been higher mean or median estimates this would have been an
indication that the test version was an improvement. Since there was no significant
positive change there was no significant information leaning towards the test version.

8

Table 4. Mean and Median Estimates of Cash Public Assistance Income
Test
Standard Control
Standard
Test Standard
Measure
Estimate
Error
Estimate
Error
Control
Error
Significance
Mean
$2,072
$182
$2,069
$134
$3
$226
No
Median
$1,713
$55
$1,664
$42
$49
$72
No
Source: U.S. Census Bureau, 2010 American Community Survey Content Test, September to December
2010

5.5 Do the changes to the cash public assistance question lower the item missing data
rates?
The item missing data rates were compared between the control and test versions for PA
recipiency and amount to see whether the control version rates are significantly higher.
The recipiency item missing data rate for the test and controls panels were computed. The
difference in the recipiency item missing data rates was also calculated. A one-sided test
was used to determine whether there is a statistically significant negative difference
between the test and control recipiency item missing data rates, using a significance level
of α = 0.10.
Table 5 shows item missing data rates for the test and the control and the difference
between the test and control for both the recipiency and the amount questions. The
changes to the question do not significantly lower the item missing data rates for either
recipiency or amount. Instead, the item missing data rates for amount are significantly
higher for the test version than control.
Table 5. Item Missing Data Rates
Test
Standard
Control
Standar Test – Control
Standard
Item Missing Data
Estimate
Error
Estimate
d Error
Estimate
Error
Rates
(%)
(%)
(%)
(%)
(%)
(%)
Recipiency
10.2
0.2
10.5
0.3
-0.3
0.4
(n=38,869)
(n=38,892)
Amount
18.9
2.5
14.7
1.8
4.2
3.1
(n=610)
(n=732)
Source: U.S. Census Bureau, 2010 American Community Survey Content Test, September to December
2010
1
Test is significantly greater than control at the α = 0.10 significance level using a one-sided test.

5.6 Do the changes to the cash public assistance question lower response error (i.e.,
bias) in the estimates of cash public assistance recipiency and cash public assistance
income?
Using data from the Content Test and CFU, we compared net difference rates (ndr)
between the control and test versions. A response was required in both survey measures
to be included in this analysis. The net difference rate provides an approximate measure
of bias in the content test estimates when we assume that the reinterview provides a
measure of “truth.” A negative NDR means that there is an overestimate of the true
values while a positive ndr means there is an underestimate.
9

Test signif.
less than
control?
No
No1

See Figure 1. Below.
Note that CFU used questions from the CPS Annual Social and Economic Supplement
for the Cash Public Assistance questions as well as the other income questions changed
for the Content Test. The CFU was identical for the Control and Test versions.
Figure 1.
CFU response
(reinterview)
Yes
No
Total

Yes
a
c

Content test response
No
Total
b
a+b
d
c+d

a+c

b+d

n = a+b+c+d

The universe for CFU is all persons age 15+ (the same as the universe for the content test
Cash Public Assistance questions). In order for estimates of ndr to be representative of
this universe, note that the elements a, b, c, and d in the table above will be sums of the
appropriate sample weights for cases (not unweighted counts).
Table 6 shows the net difference rates for recipiency and nine income ranges for the test
and control questions. Additionally, the differences between the NDRs for the test group
and control group were tested using a one-sided test with = 0.10. The difference
between the absolute values of the amount NDRs for test group and the control group
were tested using a one-sided test with a Bonferonni-Holm adjusted alpha controlling the
family-wise error level of 0.10.
The changes to the cash public assistance question significantly lower the recipiency
NDR which means the test question has reduced the overestimate of recipiency. This
outcome is the opposite of what the test version was aiming at accomplishing, therefore
producing negative results. However, the test version amount NDRs is not significantly
lower for any of the nine amount categories.

10

Table 6. Net Difference Rates

Category
Recipiency:
NDR
Amount:

NDR
Test
Estimate
(%)
(n=18,731)
-0.2
(n=436)

Standard
Error
(%)
0.1

NDR
Control
Estimate
(%)
(n=18,592)
-0.5
(n=511)

Standard Error
(%)

|Test||Control|
(%)

Standard
Error
(%)

Test signif.
less than
control?

0.1

-0.3

0.2

Yes

6.6
3.0
3.1
1.2
3.5
3.3
DK/REF/OTHER
$0
-28.3
2.9
-36.3
3.2
-8.0
4.5
$1 or $2
0.0
0.4
0.0
0.3
0.0
0.5
$3 - $199
1.8
1.5
2.8
0.8
-0.9
1.6
$200 - $499
2.5
0.9
4.1
1.7
-1.6
1.8
$500 - $999
1.3
1.7
5.5
2.8
-4.2
3.1
$1,000 -$4,999
8.9
2.1
14.7
2.4
-5.8
3.0
$5,000 -$9,999
4.0
1.9
4.7
1.9
-0.7
2.8
$10,000 or more
3.2
1.2
1.5
1.2
1.7
1.7
Source: U.S. Census Bureau, 2010 American Community Survey Content Test, September to December
2010

5.7 For each mode of data collection, do the changes to the cash public assistance
question affect the item missing data rates, the estimates of recipiency and cash
public assistance income, or the response error (i.e., bias)?
For each mode (mail, CATI, CAPI) and also for combined CATI/CAPI, the item missing
data rates were compared, estimates of recipiency and Cash Public Assistance income
and response error (i.e., bias) were calculated as above between the control and test
versions.
Table 7 show Recipiency Net Difference Rates by Mode of Interview for the Test and
Control versions and the difference between the test and control versions. Statistical
significance of differences is determined at the α = 0.10 significance level using a onesided test.
The test version resulted in a significantly lower NDR in the estimate of public assistance
recipiency for mail responses compared to control. The item missing data rate for the test
version was lower for the mail mode than the control. All other measures by mode of data
collection showed no statistical differences.

11

No
No
No
No
No
No
No
No
No

See tables A-1 to A-5 in appendix A for additional testing.
Table 7. Recipiency Net Difference Rates by Mode of Interview
Test
Standard
Control
Standard
TestEstimate
Error
Estimate
Error
Control
Mode
(%)
(%)
(%)
(%)
(%)
Mail

Standard
Error
(%)

Test signif.
less than
control?
Yes

0.2
0.1
0.8
0.1
-0.5
0.2
(n=12,804)
(n=12,710)
CATI/CAPI
0.1
0.2
0.3
0.2
-0.1
0.3
No
(n=5,927)
(n=5,882)
CATI
0.2
0.2
0.1
0.3
0.2
0.3
No
(n=2,344)
(n=2,391)
CAPI
0.1
0.3
0.3
0.3
-0.2
0.4
No
(n=3,583)
(n=3,491)
Source: U.S. Census Bureau, 2010 American Community Survey Content Test, September to December
2010.

5.8 For each mail response stratum, do the changes to the cash public assistance
question affect the item missing data rates, the estimates of recipiency and cash
public assistance income, or the response error (i.e., bias)?
For each mail response stratum, the item missing data rates were calculated, estimates of
recipiency and cash public assistance income and net difference rates were also
calculated as above.
Net difference rates for cash public assistance income recipiency for each mail response
stratum were significantly lower in the test than control. The median estimate for cash
public assistance income was significantly higher for the test version than control in the
low response stratum.

12

See tables A-6 to A-10 in Appendix A for more testing.
Table 8a. Net Difference Rates -High Response Stratum
NDR
Test
Estimate
(%)

Standard
Error
(%)

NDR
Control
Estimate
(%)

Standard
Error
(%)

|Test||Contr
ol|
(%)

Standar
d Error
(%)

Test
signif. less
than
control?

Category
Recipiency:
NDR
0.1
0.2
0.4
0.2
-0.4
0.2
Yes
Amount:
DK/REF/OTHER
9.4
4.5
1.7
1.6
7.8
4.8
No
$0
24.4
4.0
32.4
4.2
-8.0
6.1
No
$1 or $2
0.0
0.0
0.0
0.0
0.0
0.0
No
$3 - $199
2.0
2.0
2.5
1.0
-0.4
2.3
No
$200 - $499
1.6
0.9
3.3
2.1
-1.8
2.2
No
$500 - $999
0.9
2.2
5.0
3.8
-4.0
4.0
No
$1,000 -$4,999
6.4
2.6
14.6
3.0
-8.1
3.8
No
$5,000 -$9,999
2.6
2.2
4.2
2.4
-1.6
3.3
No
$10,000 or more
3.2
1.7
1.1
1.7
2.1
2.4
No
Source: U.S. Census Bureau, 2010 American Community Survey Content Test, September to December
2010
Table 8b. Net Difference Rates -Low Response Stratum

Category
Recipiency:
NDR
Amount:
DK/REF/OTHER
$0
$1 or $2
$3 - $199
$200 - $499
$500 - $999
$1,000 -$4,999
$5,000 -$9,999
$10,000 or more

NDR
Test
Estimate
(%)

Standard
Error
(%)

NDR
Control
Estimate
(%)

Standard
Error
(%)

|Test||Contr
ol|
(%)

Standar
d Error
(%)

Test
signif. less
than
control?

0.4

0.2

0.8

0.2

-0.3

0.2

Yes

1.5
35.4
0.0
1.4
4.1
5.3
13.4
6.6
3.1

0.8
3.5
0.0
1.8
1.6
2.4
3.2
3.3
1.1

6.5
45.0
0.0
3.3
5.9
6.6
14.9
5.6
2.2

2.2
3.6
0.0
1.1
1.9
2.5
3.6
2.7
0.8

5.0
9.6
0.0
2.0
1.7
1.3
1.5
1.0
0.9

2.6
4.6
0.0
2.1
2.2
3.3
4.7
4.1
1.4

No
No
No
No
No
No
No
No
No

Source: U.S. Census Bureau, 2010 American Community Survey Content Test, September to December
2010

13

Table 8c. Means and Medians- Low Response Stratum

Measure

Test
Estimate

Standard
Error

Control
Estimate

Standard
Error

Test Control

Standard
Error

Significance

Mean
$2,558
$154
$2,579
$154
$-20.24
$229
No
Median
$1,968
$80
$1,838
$59
$130
$101
YES
Source: U.S. Census Bureau, 2010 American Community Survey Content Test, September to December
2010

5.9 Does either question version elicit respondent or interviewer behaviors that may
contribute to interviewer or respondent error?
Results indicate that the test series does not perform as well as the control series on
interviewer behavior. Review of the behavior coder notes indicate that interviewers
frequently stopped reading at “… during the past 12 months” and often dropped the last
sentence (“Do not include …”). For respondent behavior, the test series performed better
than the control.
5.10 For the Hispanic and Black population subgroups, do the changes to the cash
public assistance question affect the estimate of recipiency, item missing data rate,
or reliability of the data?
For each population subgroup (Hispanic and Black) the item missing data rates were
compared, estimates of recipiency and Cash Public Assistance income and response error
(i.e., bias) were calculated as above between the control and test versions.
The results are mixed. See tables 10a to10d below. The recipiency rate for the test
version was statistically higher than the control for Hispanics. The test version produced
a significantly lower item missing data rate than the control version for Hispanics as well.
However, the test version also resulted in a statistically higher NDR for recipiency among
Hispanics. The median estimate of cash public assistance income was significantly
higher in the test version of the question among Blacks. The test version also resulted in
a statistically lower NDR for recipiency among Blacks.
Table 10a: Recipiency – Cash Public Assistance Income by Hispanic Origin and Race
Test
Standard
Control
Standard Difference Standard
Estimate
Error
Estimate
Error
Estimate
Error
Subgroup
(%)
(%)
(%)
(%)
(%)
(%)
Hispanic
2.2
0.3
1.8
0.2
0.5
0.3
Black

3.2

0.4

3.8

0.4

-0.6

0.5

Significance
Yes
No

Source: U.S. Census Bureau, 2010 American Community Survey Content Test, September to December
2010.
* Statistical significance of differences is determined at the α = 0.10 significance level using a one-sided
test.

14

Table 10b. Median Estimates of Cash Public Assistance Income by Hispanic Origin and Race
Test
Standard Control
Standard
Test Standard
Subgroup
Estimate
Error
Estimate
Error
Control
Error
Significance
Hispanic
$2,095
$201
$2,054
$131
$41
$254
No
Black
$1,993
$155
$1,715
$102
$278
$194
Yes
Source: U.S. Census Bureau, 2010 American Community Survey Content Test, September to December
2010.
* Statistical significance of differences is determined at the α = 0.10 significance level using a one-sided
test.

Table 10c: Item Missing Data Rates for Cash Public Assistance Income Recipiency by Hispanic Origin
and Race
Test
Standard
Control
Standard Difference Standard
Estimate
Error
Estimate
Error
Estimate
Error
Test signif.
Subgroup
(%)
(%)
(%)
(%)
(%)
(%)
less than
control?
Hispanic
7.5
0.5
9.0
0.6
-1.5
0.7
Yes
(n=7,529)
(n=7,244)
Black
11.5
0.6
12.1
0.7
-0.6
0.9
No
(n=6,407)
(n=6,322)
Source: U.S. Census Bureau, 2010 American Community Survey Content Test, September to December
2010.
* Statistical significance of differences is determined at the α = 0.10 significance level using a one-sided
test.

Table 10d: Net Difference Rates (NDR) Cash Public Assistance Income Recipiency by Hispanic Origin
and Race
Test
Standard
Control
Standard
|Test|Standard
Estimate
Error
Estimate
Error
|Control|
Error
Test signif.
Subgroup
(%)
(%)
(%)
(%)
(%)
(%)
less than
control?
Recipiency:
Hispanic
-0.8
0.3
-0.3
0.3
0.6
0.4
No1
(n=74)
(n=79)
Black
-0.3
0.5
-1.4
0.5
-1.2
0.7
Yes
(n=95)
(n=110)
Source: U.S. Census Bureau, 2010 American Community Survey Content Test, September to December
2010.
* Statistical significance of differences is determined at the α = 0.10 significance level using a one-sided
test.
1
Test is significantly greater than control at the α = 0.10 significance level using a one-sided test.

6. SUMMARY
There were unfavorable results in terms of higher item missing data rates for amounts and
higher net difference rates for recipiency for Hispanics. Despite some other positive
results, these two unfavorable results were in opposition to the original intent of the
question change and the goal of obtaining more recipiency was not met.

15

Based on the test results, it was apparent there were no clear advantages to changing the
cash public assistance question. It is recommended that the public assistance question
remain as currently asked and continue to be asked the same in all modes (CATI, CAPI
and mail).

References
Lynch, V., Resnick, D., Staveley, J., Taeuber, C. (2008) “Differences in Estimates of
Public Assistance Recipiency Between Surveys and Administrative Records”, US Census
Bureau and the Family Investment Administration, The Maryland Department of Human
Resources
RTI International (August 12, 2009). “Cognitive Testing of the American Community
Survey Content Test Items,” Research Triangle Park, NC)

Acknowledgements
The authors would like to acknowledge Mary C. Davis and Padraic Murphy for their
contributions to the statistical analysis of this report.

16

Appendix A: Tables
Table A-1. Recipiency

Measure

Test
Estimate

Standard
Error

Control
Estimate

Standard
Error

Test Control

Standard
Error

Significance

Mail
0.9
0.1
1.4
0.1
-0.5
0.1
No
CATI/CAPI
1.9
0.1
1.8
0.1
0.1
0.2
No
CATI
0.9
0.1
1.1
0.1
-0.1
0.2
No
CAPI
2.1
0.2
1.9
0.2
0.2
0.3
No
Source: U.S. Census Bureau, 2010 American Community Survey Content Test, September to December
2010

Table A-2. Item Missing Data Rates for Recipiency

Measure

Test
Estimate

Standard
Error

Control
Estimate

Standard
Error

Test Control

Standard
Error

Significance

Mail
16.2
0.3
16.6
0.4
-0.4
0.5
No
CATI/CAPI
2.7
0.3
2.89
0.2
-0.2
0.3
No
CATI
3.7
0.5
3.05
0.5
0.6
0.6
No
CAPI
2.4
0.3
2.84
0.3
-0.4
0.4
No
Source: U.S. Census Bureau, 2010 American Community Survey Content Test, September to December
2010

Table A-3. Net Difference Rates - CATI/CAPI
NDR
Test
Estimate
(%)

Standard
Error
(%)

NDR
Control
Estimate
(%)

Standard
Error
(%)

|Test||Contr
ol|
(%)

Standar
d Error
(%)

Category
Recipiency:
NDR
0.1
0.2
0.2
0.2
-0.1
0.3
Amount:
DK/REF/OTHER
17.3
7.4
9.5
3.7
7.8
8.3
$0
48.7
6.1
49.2
7.5
-0.5
8.8
$1 or $2
0.0
0.0
0.0
0.0
0.0
0.0
$3 - $199
3.7
3.8
1.1
0.8
2.6
3.8
$200 - $499
1.9
1.6
5.9
4.1
-3.9
4.3
$500 - $999
0.9
4.2
8.5
7.2
-7.6
7.3
$1,000 -$4,999
16.4
4.6
16.7
5.0
-0.2
6.8
$5,000 -$9,999
6.9
4.6
9.5
4.9
-2.6
7.0
$10,000 or more
1.4
0.9
2.0
3.0
-0.5
2.0
Source: U.S. Census Bureau, 2010 American Community Survey Content Test, September to December
2010

A-1

Test signif.
less than
control?
No
No
No
No
No
No
No
No
No
No

Table A-4. Net Difference Rates -CATI
NDR
Test
Estimate
(%)

Standard
Error
(%)

NDR
Control
Estimate
(%)

Standard
Error
(%)

|Test||Contr
ol|
(%)

Standar
d Error
(%)

Test signif.
less than
control?

Category
Recipiency:
NDR
0.2
0.2
0.1
0.3
0.1
0.3
No
Amount:
DK/REF/OTHER
20.5
9.6
1.9
1.9
18.6
9.7
No
$0
52.3
8.8
57.4
13.9
-5.1
17.1
No
$1 or $2
0.0
0.0
0.0
0.0
0.0
0.0
No
$3 - $199
0.0
0.0
3.9
3.0
-3.9
3.0
No
$200 - $499
1.8
3.2
5.8
3.6
-4.0
4.4
No
$500 - $999
7.6
7.9
10.3
9.4
-2.7
12.0
No
$1,000 -$4,999
16.9
7.6
35.5
14.1
-18.5
15.5
No
$5,000 -$9,999
3.6
2.6
8.4
8.5
-4.6
8.8
No
$10,000 or more
1.8
2.0
8.3
8.5
-6.5
8.6
No
Source: U.S. Census Bureau, 2010 American Community Survey Content Test, September to December
2010

Table A-5. Net Difference Rates -CAPI
NDR
Test
Estimate
(%)

Standard
Error
(%)

NDR
Control
Estimate
(%)

Standard
Error
(%)

|Test||Contr
ol|
(%)

Standar
d Error
(%)

Test
signif. less
than
control?

Category
Recipiency:
NDR
0.1
0.3
0.3
0.3
-0.2
0.4
No
Amount:
DK/REF/OTHER
16.9
9.0
10.5
4.2
6.3
10.1
No
$0
48.2
7.1
48.2
8.4
0.0
10.1
No
$1 or $2
0.0
0.0
0.0
0.0
0.0
0.0
No
$3 - $199
4.3
4.4
0.7
0.8
3.6
4.4
No
$200 - $499
2.0
1.8
6.0
4.5
-3.9
4.8
No
$500 - $999
0.0
4.7
8.3
8.0
-8.2
8.3
No
$1,000 -$4,999
16.3
5.2
14.3
5.0
2.1
7.3
No
$5,000 -$9,999
7.4
5.3
11.8
5.6
-4.4
8.0
No
$10,000 or more
1.4
1.0
3.3
3.2
-1.9
3.3
No
Source: U.S. Census Bureau, 2010 American Community Survey Content Test, September to December
2010

A-2

Table A-6. High Response Stratum-Recipiency

Measure

Test
Estimate

Standard
Error

Control
Estimate

Standard
Error

Test Control

Standard
Error

Significance

Recipiency
1.0
0.1
1.2
0.1
-0.2
0.1
No
Source: U.S. Census Bureau, 2010 American Community Survey Content Test, September to December
2010

Table A-7. Item Missing Data Rates- High Response Stratum

Measure

Test
Estimate

Standard
Error

Control
Estimate

Standard
Error

Test Control

Standard
Error

Significance

Recipiency
10.3
0.3
10.7
0.4
-0.3
0.5
No
Amount
9.1
0.3
9.5
0.3
-0.3
0.4
No
Source: U.S. Census Bureau, 2010 American Community Survey Content Test, September to December
2010

Table A-8. Means and Medians- High Response Stratum

Measure

Test
Estimate

Standard
Error

Control
Estimate

Standard
Error

Test Control

Standard
Error

Significance

Mean
$1,817
$268
$1,818
$201
$-1.75
$343
No
Median
$1,603
$69
$1,590
$55
$13
$92
No
Source: U.S. Census Bureau, 2010 American Community Survey Content Test, September to December
2010

Table A-9. Low Response Stratum

Measure

Test
Estimate

Standard
Error

Control
Estimate

Standard
Error

Test Control

Standard
Error

Significance

Recipiency
2.3
0.1
2.5
0.1
-0.2
0.2
No
Source: U.S. Census Bureau, 2010 American Community Survey Content Test, September to December
2010

A-3

Table A-10. Item Missing Data Rates- Low Response Stratum

Measure

Test
Estimate

Standard
Error

Control
Estimate

Standard
Error

Test Control

Standard
Error

Significance

Recipiency
9.7
0.2
10.0
0.3
-0.3
0.4
No
Amount
8.1
0.2
8.3
0.2
-0.2
0.3
No
Source: U.S. Census Bureau, 2010 American Community Survey Content Test, September to December
2010

A-4

Appendix B: Images of the Mail Versions of the Control and Test
Questions/Cognitive Testing Wording

Figure B-1. Control Version of the Cash Public Assistance Question
Current ACS Question

Figure B-2. Test Version of the Cash Public Assistance Question
Content Test Question

B-3. Cognitive Testing Wording.
Did [/you] receive any welfare payments or cash assistance from the state or
local welfare office, for [/yourself] or any children in this household
DURING THE PAST 12 MONTHS? Include all assistance, even if for only one
payment. Do NOT include benefits from food, energy, or rental assistance
programs.
<1> Yes
<2> No
[IF YES] What was the amount?
Did [/you] receive any welfare payments or cash assistance from the state or
local welfare office, for [/yourself] or any children in this household
during the past 12 months? Include all assistance, even if for only one month. Do
NOT include benefits from food, energy, or rental assistance programs.
B-1

<1> Yes
<2> No
[IF YES] What was the amount?

B-2

Appendix C: CATI and CAPI Versions of the Control and Test Questions
CONTROL Wording
Did [FILL1: /you] receive any Cash Public Assistance or welfare payments
from the state or local welfare office DURING THE PAST 12 MONTHS?
<1> Yes
<2> No
If Yes: What was the amount?
TEST Wording
Did [FILL1: /you] receive any welfare payments or Cash assistance from the
state or local welfare office, for [FILL1: /yourself] or any children in this
household DURING THE PAST 12 MONTHS? Include all assistance, even if for only
one month. Do NOT include benefits from food, energy, or rental assistance programs.

<1> Yes
<2> No
If yes: "What was the amount?"

C-1

Appendix D: Flow of the Content Follow-Up

D-1

Appendix E: Information Page
Test Design
Treatments

Two question versions with different wording (see page 4).

Sample Size

35,000 households per treatment (70,000 total)

Sample Design
Modes

Time Frame

Similar to production ACS with an additional level of stratification into high
and low mail response areas.
Mail, CATI, and CAPI, with a CATI content follow-up (CFU) of all
households. CATI and CAPI interviews will be recorded using ComputerAssisted Recorded Interviewing (CARI) technology.
Same schedule as the production September panel: mailout in late August,
CATI in October, CAPI in November. CFU goes from mid-September to
mid-December.

Research Questions & Evaluation Measures
No.
Research
Questions
1 Is the response
distribution
of cash public
assistance income comparable to the
Current Population Survey’s Annual Social
and Economic Supplement (CPS ASEC)
distribution of cash public assistance
income?

2

3

4

5

Do the changes to the cash public
assistance question raise the estimate of
cash public assistance income?
Do the changes to the cash public
assistance question lower response error
(i.e., bias) in the estimates of cash public
assistance recipiency and cash public
assistance income?
Do the changes to the cash public
assistance question lower the item missing
data rates?
Do the changes to the cash public
assistance question raise the estimate of
persons receiving cash public assistance?

E-1

Evaluation Measures
Compare the response distribution of cash
public assistance income between the test
version and the CPS ASEC.
We cannot make formal statistical
comparisons since the Content Test data
will not have been edited or imputed,
adjusted for nonresponse, nor raked to
known population totals.
Compare the mean and median estimates
of cash public assistance income between
the control and test versions.
Using data from the Content Test and
CFU, compare net difference rates between
the control and test versions (based on
answers to more detailed content follow-up
questions).
Compare the item missing data rates
between the control and the test versions.
Compare the estimate of persons receiving
cash public assistance between the control
and test versions.

No.
6 For eachResearch
mode of dataQuestions
collection, do the
changes to the cash public assistance
question affect the item missing data rates,
the estimates of recipiency and cash public
assistance income, or the response error
(i.e., bias)?

7

8

9

For each mail response stratum, do the
changes to the cash public assistance
question affect the item missing data rates,
the estimates of recipiency and cash public
assistance income, or the response error
(i.e., bias)?
Does either question version elicit
respondent or interviewer behaviors that
may contribute to interviewer or
respondent error?
For the Hispanic and Black population
subgroups, do the changes to the cash
public assistance question affect the
estimate of recipiency, item missing data
rate, or reliability of the data?

Evaluation Measures
For each mode (mail,CATI,CAPI),
compare the item missing data rates,
estimates of recipiency and cash public
assistance income, and response error (i.e.,
bias) between the control and the test
versions.
Comparisons across modes of data
collection cannot be made since
measurable differences cannot be
attributed strictly to the mode of data
collection. Observed differences across
modes may also be due to mode specific
respondent characteristics and reinterview
mode effects (CFU only).
For each mail response stratum (high and
low), compare the item missing data rates,
estimates of recipiency and cash public
assistance income, and response error (i.e.,
bias) between the control and the test
versions.
Compare the behavior coding results
derived from the CARI recordings between
the control and the test versions.
For the Hispanic and Black subgroups
separately, compare the item missing data
rates, estimates of cash public assistance
recipiency, and reliability measures
between the control and the test versions.
Note: This test was not designed to study
differences across panels by race/ethnicity
breakdowns with statistical precision, as
this was not a stated goal of the test.
Therefore, these results will be provided
for informational purposes only.

E-2

Selection Criteria (In order of priority)

Research
Question(s)

Criteria

1

The overall distribution of cash public assistance income for the test
version should be comparable to that of the CPS ASEC.

2-3

An increase in cash public assistance receipt and the amount of cash
public assistance received in the test version implies a positive
change since this item is historically underestimated

4-5

The item missing data rates and response error (i.e., bias) will be
considered together when determining whether the test version
performs better.

Supplemental Information
Research
Criteria
Question(s)
6-9
Not part of the selection criteria. These data are presented to give additional
information regarding how the questions performed.

E-3

Appendix F: CFU wording

CPS INSERT FOR CASH PUBLIC ASSISTANCE AND WELFARE

CPS Q59A88
At any time in the past 12 months, even for one month, did receive any CASH
assistance from a state or local welfare program (if possible ?
Do not include food stamps, SSI, energy assistance, WIC, School meals, or transportation,
childcare, rental, or education assistance.
-(Display this information on the interviewer question screenInclude Cash payments from:
welfare or welfare-to-work programs, (State Program Name and/or acronyms),Temporary
Assistance for Needy Families program (TANF),Aid to Families with Dependent Children
(AFDC),General Assistance/Emergency Assistance program, Diversion Payments, Refugee Cash
and Medical Assistance program, General Assistance from Bureau of Indian Affairs, or Tribal
Administered General Assistance)
1 Yes (skip to CPS Q59C8)
2 No

CPS Q59A89
Just to be sure, in the past 12 months, did receive CASH assistance from a state or
local welfare program, on behalf of CHILDREN in the household?
1 Yes
2 No (skip to ACS 47ga)

CPS Q59C8
From what type of program did (name/you) receive the CASH assistance? Was it a welfare or
welfare-to-work program such as (State Program Name), General Assistance, Emergency
Assistance, or some other program?
- Enter all that apply, separate with commas
- Probe: Any Other Program?
1 (State Program Name)/welfare/AFDC
2 General Assistance
3 Emergency Assistance/short-term Cash assistance
4 Some other program (specify)

CPS Q59C8s
What type of program?

F-1

CPS Q59e
In the past 12 months, how much CASH assistance did (name/you) receive?
- Enter dollar amount

F-2