ASEC Redesign and Evaluation Project Plan for Income

Attachment K

2013 ASEC Redesign TEST

Ed Welniak and Jessica Semega (SEHSD)

January 29, 2014

Introduction

The Census Bureau conducted the 2013 Current Population Annual Social and Economic Supplement (ASEC) Redesign Test to evaluate proposed changes to the content. The objective, for both new and existing questions, was to determine the impact of changing question wording, response categories, and redefinition of underlying constructs on the quality of data collected. This document contains results related to then test.

Test Design

Treatments	Two questionnaire versions; Test conducted in CATI; Control conducted in CATI/CAPI
Sample Size	9,000 households Test, 77,000 Control
Sample Design	Similar to production ACS with an additional level of stratification into high and low mail response areas.
Modes	CATI, and CAPI
Time Frame	Test conducted in weeks 1, 2, and 4; Control was Feb-April production schedule

Justification for Question Change

Research shows the ASEC suffers from misclassification and underreporting of income. The largest aggregate shortfalls in measured CPS income are in asset income and means-tested transfer income. Contributing to the shortfall is the lack of information on lump-sum payments. This is potentially a very large problem for pension income and to a lesser extent disability payments that are typically paid as lump sums.

Summary of Findings

The proposed changes to the ASEC instrument significantly increased the proportion of people age 15 and older reporting recipiency of:

1. Interest

2. Dividends

3. Survivor

4. Disability and

5. Retirement/Pension

Estimate of aggregate income for:

1. Total money income

2. Interest

3. Dividends

4. Disability and

5. Retirement/Pensions

Main Research Questions & Evaluation Measures

No.	Research Questions	Evaluation Measures	Results
1	Does the redesigned questionnaire raise the proportion of people receiving: Interest Dividends Survivor Disability Pension/Retirement Public Assistance	response distributions	The changes to the questionnaire increased recipiency for all but public assistance where there was a statistically significant decline of 0.3 percentage points. (See Table 1.)
2	Do the changes to the income questions raise the estimate of income for: Total money income Interest Dividends Survivor Disability Pension/Retirement Public Assistance	Aggregate income	The changes to the questionnaire increased aggregate income for all but survivor income and public assistance where there was no statistically significant difference. (See Table 2.)

Table 1. Recipiency Rates
Income Source	Test Estimate (%)	Control Estimate (%)	Test - Control (%)	Standard Error (%)
1. Interest	45.8	28.3	*17.5	0.37
2. Dividends	20.5	11.3	*9.2	0.26
3. Survivor	1.8	0.9	*0.95	0.08
4. Disability1	2.6	0.5	*2.2	0.07
5. Pension/Retirement	42.9	5.3	*37.7	0.23
6. Public Assistance	0.4	0.7	*-0.3	0.06
Source: U.S. Census Bureau, 2013 ASEC Redesign Content Test, March 2013
* Statistically significant at the α = 0.10 level.
1 Disability does not include Social Security Disability Insurance (SSDI)
Table 2. Aggregate Income
Income Source	Test Estimate ($ billions)	Control Estimate ($ billions)	Percent Difference (Test/Control)	Standard Error (%)
1. Total Money Income	8,600	5,967	*44.1	5.2
2. Interest	294.4	42.2	*597.6	66.1
3. Dividends	105.5	49.6	*112.6	27.4
4. Survivor	33.9	29.3	15.6	17.6
5. Disability	99.4	15.1	*556.1	77.9
6. Pension/Retirement	341.0	241.1	*41.4	8.2
7. Public Assistance	4.8	4.4	9.5	24.0
Source: U.S. Census Bureau, 2013 ASEC Redesign Content Test, March 2013
* Statistically significant at the α = 0.10 level.

3