Download:
pdf |
pdfU.S. Census Bureau
SIPP-EHC
2011 and 2012 Field Test Evaluation
September 2013
The Survey of Income and Program Participation
Principal Investigator
David Johnson, Chief, Social, Economic, and Housing Statistics Division, U.S. Census Bureau
Survey Director
Jason Fields
Contributors to the Report
Kurt Bauman
Megan Benetsky
Matthew Brault
Rebecca Chenevert
Jamie Choi
Tyler Crabb
Ralph Culver III
Judy Eargle
Ashley Edwards
Renee Ellis
C. Sol Espinoza
Stephanie Ewert
Alison Fields
Graton Gathright (Editor)
Katherine Giefer
Al Gottschalck
Timothy Grall
John Hisnanick
Shelley Irving
Hubert Janicki
Lynda Laughlin
Tracy Loveless
Stephen Kingsbury
Rose Kreider
Peter Mateyka
Brett O’Hara
Daniel Perez-Lopez
Lori Reeder
Trudi Renwick
Erik Scherpf
Jeremy Skog
Adam Smith
Amy Steinweg
Martha Stinson
Jamie Taber
Marina Vornovytskyy
Christopher Wignall
Kelly Wilkin
This report is produced to inform interested parties of ongoing Census Bureau research. The results have
been formally reviewed to ensure no confidential data have been disclosed.
Contents
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
Executive Summary ........................................................................................................................
Introduction.......................................................................................................................................
Methods and Data ...........................................................................................................................
Demographics ..................................................................................................................................
Asset Ownership..............................................................................................................................
Child Support ...................................................................................................................................
Disability............................................................................................................................................
Education ..........................................................................................................................................
Employment and Earnings .............................................................................................................
Health Insurance..............................................................................................................................
Household Composition..................................................................................................................
Housing Subsidies...........................................................................................................................
Medicaid............................................................................................................................................
Medicare ...........................................................................................................................................
Nativity and Citizenship ..................................................................................................................
Old-Age, Survivors, and Disability Insurance ..............................................................................
Poverty ..............................................................................................................................................
Residence.........................................................................................................................................
Supplemental Nutrition Assistance Program ...............................................................................
Supplemental Security Income......................................................................................................
Temporary Assistance for Needy Families...................................................................................
Unemployment Insurance...............................................................................................................
Transitions and Seams....................................................................................................................
Works Cited ......................................................................................................................................
1
7
10
16
19
24
27
29
33
41
46
50
53
58
62
65
71
74
77
82
87
91
95
99
Executive Summary
The 2014 panel of the Survey of Income and Program Participation will employ a completely re-engineered
survey instrument that includes an Event History Calendar (EHC) structured around a one-year reference
period for each interview. The new format is designed to reduce survey costs and respondent burden without
sacrificing data quality.
This report evaluates data from a two-wave field test of the re-engineered instrument (SIPP-EHC) by
comparison with data from the traditional question list instrument (SIPP) and with administrative records.
With very few exceptions, agreement between survey and administrative data is higher for SIPP-EHC or
not statistically different between surveys. While estimates from the two surveys (SIPP-EHC and SIPP) do
differ statistically in many cases, these differences are typically small and correspond to rates of agreement
with administrative data that are not lower for SIPP-EHC than for SIPP. There is little evidence that key
estimates from SIPP-EHC data are less accurate for periods earlier in the one-year reference period as might
be expected it respondents had difficulty reporting events further in the past. However, reported transitions
in program participation or other status do tend to fall disproportionately at the beginning of reference
periods. It also appears that this bias in the measurement of transitions can be improved by using information
from prior waves in interviewing and editing.
The SIPP-EHC, like SIPP, collects information about a variety of topics, including employment, income,
participation in various government programs, health insurance coverage, and demographics. This report
evaluates survey estimates for nineteen SIPP-EHC topics: assets, child support, disability, education, employment and earnings, health insurance, household composition, housing subsidies, Medicaid, Medicare,
migration, nativity and citizenship, Old-Age Survivors and Disability Insurance (OASDI), poverty, Supplemental Nutrition Assistance Program (SNAP), Supplemental Security Income (SSI), Temporary Assistance
for Needy Families (TANF), and unemployment insurance.
The SIPP-EHC, like SIPP, uses a panel design which collects monthly information by regularly interviewing
respondents about the recent past. The SIPP-EHC is structured around one-year reference periods, the SIPP
around four-month reference periods. The SIPP-EHC instrument visually represents the reference period,
and interviewers are encouraged to reference landmark events to aid respomdents recall. This calendar
design is intended to work with the structure of autobiographical memory to improve accuracy of reporting
about the longer reference period.
A longer reference period raises two possible data quality issues addressed in this report: reverse telescoping
and seam bias. Reverse telescoping refers to less precise reporting about events further in the past. Seam bias
is the disproportionate incidence of transitions in status at the beginning of reference periods. Transitions in
status refer to the reported beginning or ending of a spell of program participation, employment, educational
enrollment, health insurance coverage, or other activity.
Methods
This report evaluates estimates of key population characteristics for the nineteen topics listed above. Estimates are compared across SIPP-EHC and SIPP for calendar years 2010 and 2011. For topics for which
administrative records are available, mean absolute deviations are calculated between the survey responses
and administrative records and are compared across surveys and years. Mean participation rates are calculated for all programs. Where appropriate, mean values of amounts are provided. Medians are presented for
asset values and employment earnings to limit the influence of outliers on the comparisons. Annual rates,
number of spells, per person and number of transitions per person are also studied.
1
2
Executive Summary
The sample for the SIPP-EHC field test is not nationally representative, but was designed to include persons
likely to be participating in government programs. In this evaluation, the comparison data from 2008 SIPP
for CY2010 and CY2011 is appropriately restricted to provide a valid benchmark for the SIPP-EHC field
test. Attrition from 2008 SIPP could affect the comparability of the samples and person-month observations
between SIPP-EHC and SIPP.
For the analysis in this report, the data from the two surveys are harmonized to allow valid comparisons.
Preliminary editing of the raw SIPP-EHC survey responses is performed to create variables comparable to
those in edited SIPP data. However, no imputation of missing data in SIPP-EHC was performed, so, for
comparability, all imputed data in SIPP is excluded from this analysis. Consequently, differences in the
amount of missing data between SIPP-EHC and SIPP could explain some observed differences or mask
differences not detected in this analysis. Attrition from 2008 SIPP could affect the comparability of the
sample of person-month observations from 2008 SIPP to the sample from SIPP-EHC. The differences between
the surveys for child support, housing subsidy, and employment could be explained by the differences
in item-nonresponse rates between the surveys for these topics. In particular, sample persons without
employment may be more likely to be coded as missing in SIPP-EHC than in SIPP since the former required
respondents to positively report spells of non-employment. Additional data harmonization details are
discussed in each topic chapter.
This report makes use of administrative records for the following topics: employment and earnings, Medicare,
Medicaid, OASDI, SNAP, SSI and TANF. These data are linked to survey data using standard Census Bureau
linking processes.
Results
Table 1 presents evidence that while the survey estimates typically differ statistically, in almost all cases
the SIPP-EHC is not found to be less accurate than SIPP. The table summarizes comparisons between the
two surveys and between the surveys and administrative records. The columns labeled “Survey estimates
differ” indicate with a check mark the topics for which a t-test rejects the hypothesis that the estimates in
the surveys are equal. The columns labeled “SIPP-EHC at least as accurate as SIPP” indicate with a check
mark the topics for which the rate of disagreement with administrative data for SIPP-EHC is lower or not
statistically different from SIPP.
Table 2 presents monthly estimates for the two surveys for CY2010 and CY2011 of rates of participation in
various programs, rates of ownership of specific types of assets, rates of child support receipt, and rates of
conditions such as poverty or disability.
T-statistics greater than 1.96 are taken to indicate that the difference in the estimates for the indicated year
the two surveys is statistically significant. Table 3 presents estimates for the two surveys for CY2010 and
CY2011 of rates of participation in various programs, rates of ownership of specific types of assets, rates
of child support receipt, and rates of conditions such as poverty or disability. None of the differences in
estimates are large enough to indicate a problem with the SIPP-EHC data.
Table 4 presents means of reported dollar amounts related to income, assets, and program benefits. The
difference in median retirement savings between SIPP-EHC and SIPP could be explained by the lower
item-nonresponse rate about amount of retirement savings in SIPP-EHC. If the marginal retirement savings
responses captured in SIPP-EHC are more likely to be about low balances, this would produce the observed
difference. The difference in UI amounts in CY2011 is anomalous and being investigated.
The analysis in this report also looks for evidence of reverse telescoping, the possibility that respondents
will report less accurately about events further in the past. There is mixed evidence of such problems in
3
the pattern of agreement with administrative records for monthly indicators. Rates of under-reporting of
program participation show almost no evidence of this patters (only Medicaid in CY2010 shows this patterns).
Rates of over-reporting of program participation do tend to decline over the course of each reference year for
some topics (Medicaid, Medicare, OASDI, and SNAP).
There is evidence that transitions in status tend to fall disproportionately on the seam between the SIPPEHC waves, in January 2011. However, incorporating information from the prior interview in dependent
interviewing appears to mitigate the problem somewhat. There is also evidence from initial investigation of
reported Medicaid transitions, that data can be edited based on information from the previous interview to
increase accuracy and reduce seam bias in the edited data.
Executive Summary
4
Table 1: Accuracy of SIPP-EHC reporting: administrative records comparison
CY 2010
Estimates
differ
SIPP-EHC
accurate
Estimates
differ
SIPP-EHC
accurate
Reference
number of employers
X
X
X
X
Table 8.2
mean annual earnings
X
X
X
X
Table 8.8
monthly participation
·
X
X
X
Table 12.1
annual participation
X
X
X
X
Table 12.2
months of participation
X
X
X
X
Table 12.2
number of spells
X
X
Table 12.3
number of CY2011 transitions
X
X
Table 22.1
Topic
Variable
Employment
Medicaid
Medicare
OASDI
CY 2011
monthly participation
X
X
·
X
Table 13.1
annual participation
X
X
X
X
Table 13.2
months of participation
X
X
X
X
Table 13.2
number of spells
X
X
Table 13.3
number of CY2011 transitions
X
X
Table 22.1
monthly participation
X
X
X
X
Table 15.1
annual participation
X
X
X
X
Table 15.2
months of participation
X
X
X
X
Table 15.2
amounts
·
X
X
X
Table 15.3
retirement
·
X
·
X
Table 15.4
disability
·
X
·
X
Table 15.4
·
X
Table 15.5
number of spells
SNAP
SSI
TANF
monthly participation
X
X
X
X
Table 18.1
annual participation
X
X
X
X
Table 18.2
months of participation
X
X
X
X
Table 18.2
amounts
X
X
·
X
Table 18.3
number of spells
X
X
Table 18.4
number of CY2011 transitions
X
X
Table 22.1
monthly participation
X
·
X
·
Table 19.1
annual participation
X
X
X
X
Table 19.2
months of participation
X
X
X
X
Table 19.2
amounts
X
·
X
·
Table 19.3
number of spells
X
X
Table 19.4
number of CY2011 transitions
·
X
Table 22.1
·
X
Table 20.1
X
X
Table 22.1
monthly participation
number of CY2011 transitions
·
·
Columns labeled “Estimates differ” indicate with a Xwhether a t-test rejects the hypothesis that SIPP-EHC and SIPP estimates have
equal mean values for the indicated variables. For the CY2011 transitions rows, the comparison of the survey estimates is made only for
sample-persons for whom administrative records are available.
Columns labeled “SIPP-EHC accurate” indicate with a Xwhether the rate of disagreement (mean absolute deviation) between the
survey and administrative data for SIPP-EHC is lower or not statistically different than for SIPP.
For spell count rows, the comparisons reported in the CY2011 column are for both years combined.
5
Table 2: Summary of results: monthly participation rates
CY 2010
Variable
CY 2011
Reference
SIPP
EHC
t-stat
SIPP
EHC
t-stat
Child support receipt
0.23
0.14
4.50
0.22
0.15
3.58
Table 5.1
Employment
0.48
0.57
8.25
0.48
0.57
8.78
Table 8.1
School enrollment
0.15
0.14
2.75
0.15
0.17
3.15
Table 7.2
Health insurance coverage
0.42
0.37
7.13
0.42
0.39
3.70
Table 9.1
Housing subsidy
0.24
0.18
7.98
0.23
0.16
7.79
Table 11.1
Medicaid participation
0.33
0.33
0.28
0.32
0.35
3.92
Table 12.1
Medicare participation
0.17
0.15
3.09
0.17
0.17
0.08
Table 13.1
OASDI participation
0.15
0.12
6.09
0.16
0.14
2.32
Table 15.1
OASDI retirement
0.61
0.64
1.29
0.63
0.65
1.05
Table 15.4
OASDI disability
0.26
0.27
0.39
0.26
0.29
1.43
Table 15.4
Relocation
0.02
0.02
3.97
0.01
0.01
2.01
Table 17.2
SNAP participation
0.27
0.21
8.45
0.27
0.24
4.05
Table 18.1
SSI participation
0.05
0.08
6.06
0.05
0.09
7.42
Table 19.1
TANF participation
0.03
0.03
1.60
0.03
0.03
0.02
Table 20.1
UI benefits
0.03
0.03
0.15
0.02
0.02
2.64
Table 21.1
Table 3: Summary of results: annual participation rates
CY 2010
Variable
CY 2011
Reference
SIPP
EHC
t-stat
SIPP
EHC
t-stat
Interest-bearing assets
0.36
0.34
2.13
0.39
0.35
3.66
Table 4.1
Retirement savings
0.17
0.14
4.63
0.17
0.14
4.55
Table 4.1
Disabled
0.21
0.21
0.12
0.18
0.22
5.31
Table 6.1
Poverty
0.38
0.37
1.34
0.40
0.41
0.57
Table 16.1
Relocation
0.12
0.20
11.68
0.10
0.13
5.92
Table 17.2
UI benefits
0.05
0.05
0.45
0.04
0.03
1.33
Table 21.1
Executive Summary
6
Table 4: Summary of results: amount variables
CY 2010
Variable
CY 2011
Reference
SIPP
EHC
t-stat
SIPP
EHC
t-stat
Median retirement savings
$63,202
$15,667
5.02
$68,907
$18,000
1.43
Table 4.2
Child support amount
$406.55
$375.86
0.57
$373.78
$396.44
0.40
Table 5.2
OASDI benefits
$848.34
$899.25
1.26
$840.60
$940.72
2.78
Table 15.3
Median earnings
$1,944
$1,964
1.37
$1,947
$2,033
4.45
Table 8.6
SNAP benefits
$301.52
$353.92
3.70
$279.83
$298.52
1.50
Table 18.3
SSI benefits
$553.57
$669.16
5.69
$558.47
$732.40
5.40
Table 19.3
TANF benefits
$406.31
$370.57
0.94
$412.14
$420.07
0.13
Table 20.3
UI benefits
$900.43
$935.72
0.34
$932.41
$623.25
3.54
Table 21.2
1. Introduction
This report presents an evaluation of the new survey instrument that has been developed for the 2014 redesign
of the Survey of Income and Program Participation. The analysis compares survey measures between the
field tests of the redesigned survey instrument (SIPP-EHC) and the current production instrument (SIPP) for
concurrent periods. For several topics, linked administrative records are used to compare misreporting in
the two surveys. Topics analyzed include employment, earnings, health insurance coverage, assets, child
support receipt, an array of federal and state program participation, and characteristics of persons and
households including demographics, migration, and poverty.
A focus of this report is to compare key estimates between SIPP-EHC and SIPP. Another focus is to investigate
the impact on the survey estimates of the primary innovations in the redesign: a one-year reference period
(changed from a four-month period in the previous design) and the incorporation of an event-history
calendar for much of the survey content. In particular, the analysis investigates whether reporting about
events less temporally proximate to the time of interviewing is less precise.
This introduction provides background information on the SIPP, the new SIPP redesign, and literature related
to this evaluation. The next chapter discusses the methodology and data employed in the analysis presented
in this report. The subsequent 19 chapters discuss the data and analysis results for particular SIPP topics.
The final chapter presents an analysis of reporting of transitions in program participation, employment, and
school enrollment.
The Survey of Income and Program Participation
The Survey of Income and Program Participation1 is sponsored by the U.S. Census Bureau under the
authority of Title 13, United States Code, Section 182. The purpose of the survey is to collect reports
of income amounts, labor force activity, program eligibility and participation, and general demographic
characteristics to permit measurement of the effectiveness of federal, state, and local government programs;
to estimate future costs and coverage for government programs; and to provide improved statistics on the
distribution of income and measures of economic well-being in the country.
The survey design is a continuous series of national panels. The sample is a multistage-stratified sample of
the U.S. civilian non-institutionalized population, with sample size ranging from approximately 14,000 to
45,000 interviewed households, depending on the panel. The first panel of the survey, called the 1984 panel,
began in October 1983. Complete panels have interviewed sampled households every four months for two
to six, depending on the panel. The current panel, called the 2008 panel, began in 2008, with interviewing
every four months for a total of six years.
All adult household members (age 15 year or older) are interviewed by self-response if possible; proxy
interviews are conducted for household members aged less than 15 years or who are not available for
interviewing. Interviews are conducted by personal visit or by telephone. Beginning with the 1996 panel,
SIPP interviews have been conducted using a computer-assisted interview.
The 2014 panel will begin interviewing early in 2014. The sample for the panel is expected to include 52,000
households. Interviewing for the 2014 panel will have a one year reference period for each interview, with
calendar year 2013 as the first reference period. It is anticipated that the panel will last for four one-year
waves.
1 Detailed
information about the survey is available at http://www.census.gov/sipp/
7
8
Introduction
Re-engineering of SIPP
In 2006, the U.S. Census Bureau began a complete redesign of the Survey of Income and Program Participation. See National Research Council (2009) and Moore, et al. (2009) for additional background on SIPP
and the redesign. The goals of the redesign are to reduce costs and respondent burden and to improve
data quality and timeliness. The new survey instrument, called SIPP-EHC, is scheduled to become the
production survey instrument for the SIPP program beginning in 2014. The SIPP-EHC instrument is a
complete redevelopment in Blaise and C# of the previous SIPP survey instrument that was implemented in
CASES. The instrument is built around the change in survey reference period for the 2014 panel from four
months to one year. The SIPP-EHC incorporates an event history calendar design to help ensure that the
2014 panel will continue to collect intra-year dynamics of income, program participation, and other activities
with at least the same data quality as earlier panels. Event history calendars are discussed in more detail in
the next section. The instrument collects information at the monthly level. A 2009 National Academy of
Sciences study describes the design of SIPP-EHC with particular emphasis on design features. See National
Research Council (2009).
Five field tests of the SIPP-EHC instrument have taken place (in 2008, 2010, 2011, 2012, and 2013). The 2012
SIPP-EHC field test is a wave 2 interview of the 2011 SIPP-EHC field test sample. The reference year for
waves one and two of the 2011 SIPP-EHC field tests were calendar years 2010 and 2011. The 2013 SIPP-EHC
field test is a wave 3 interview of this same sample. This report evaluates the 2011 and 2012 field tests.
Previous evaluations of the SIPP-EHC
This report is the most comprehensive evaluation of the SIPP-EHC to date and the present analysis evaluates
the SIPP-EHC using a broader array of administrative records than previous work. Gathright, Stinson, and
Reeder (2012) evaluated the 2010 and 2011 SIPP-EHC using a more limited set of administrative records and
did not compare SIPP-EHC to SIPP for topics for which no administrative records were available. Stinson,
Gathright, and Skog (2012) evaluated reporting about employment and earnings in 2010 SIPP-EHC using
administrative data on employment and earnings. An earlier Census Bureau working paper (SIPP-EHC Data
Evaluation Workgroup, 2011) compared program participation rates for SNAP, TANF, WIC, Medicare, and
Medicaid and employment rates between 2010 SIPP-EHC and benchmark samples from 2008 SIPP. Moore, et
al. (2009) compared responses in SIPP and SIPP-EHC surveys for a sample that was first interviewed in the
2004 SIPP panel and then re-interviewed using an early version of the SIPP-EHC instrument.
The use of administrative records for evaluation of SIPP data quality dates back to the first SIPP panel in 1984
(Marquis and Moore, 1990). Abowd and Stinson (2011) use administrative data to estimate measurement
error in SIPP annual job earnings. The methodology of the analysis in this report is related to the work of
Meyer and Goerge (2011) on misreporting of SNAP participation in the Current Population Survey and the
American Community Survey.
Literature on testing of event history calendars
EHC instruments are designed to work with the structure of autobiographical memory to improve the
accuracy of survey reporting. An EHC is designed to support the respondent’s ability to accurately recall
changes over a reference period by providing visual representation of the relevant months and event cues.
An EHC has three components: a visualization of the reference period, generally subdivided into smaller
units of time; cues relating to themes of interest; and temporal boundaries defined by landmark events
(Glasner and van der Vaart, 2009). Calendar methods are generally used to examine extended reference
periods. Life course research may design instruments to cover the reference period from the respondent’s
birth to the time of the interview. That frame provides a very different set of challenges than designing an
instrument to cover a calendar year or cohort, but an EHC instrument can be effective for covering relatively
Chapter 1
9
brief reference periods by creating relevant landmarks and dividing the period into spells (Axinn, Pearce,
and Ghimire, 1999). The provision of landmark events, reference points, and spells allows a respondent to
define more specific periods of time within a larger reference period.
Theories from cognitive science and previous evaluations both support the merits of an EHC for improving
data quality. Though many previous evaluations have been undertaken, none have been as extensive as
the SIPP-EHC field tests. EHC methods are supported by the move toward understanding the cognitive
underpinnings of survey design (Tourangeau 2000). A calendar, providing temporal and thematic queues, is
thought to work better with the structure of autobiographical memory. Memory is relational and dependent
upon contextual clues for accurate recall. Traditional questionnaire designs that attempt to segment off
particular topics or periods work against the structure of memory, particularly for events which have
recurred over a period of time (Belli 1998).
An EHC offers flexibility at the expense of standardization, so interviewer effects may be magnified by this
format. Interviewer experience was controlled for through matched pair random assignment (interviewers
were paired based on years experience, and one randomly assigned to the EHC and the other to the Q-List
condition), demonstrating that a slightly stronger interviewer effect is present in EHCs (Sayles, Belli, and
Serrano, 2010). Seam effects were found to be decreased in EHCs over a reference period of two years,
however a “false seam” near the middle of the reference period developed (Callegaro, 2007).
The SIPP-EHC field tests have provided the opportunity to empirically test the effectiveness of EHCs
through comparison with administrative records. Previous direct comparisons between EHC instruments
and traditional question list (Q-List) formats have generally relied upon reinterviewing respondents from
longitudinal studies. For instance, Belli, Shay and Stafford (2001) took a random sample of respondents from
The Panel Study of Income Dynamics (PSID) and reinterviewed them in 1998 about the previous two years.
Respondents and interviewers were randomly assigned to either a Q-List or EHC condition. The results
were compared to the responses those respondents had given in the 1997 PSID interview about the events
of 1996, assuming that the information collected nearer to the reference period would be more accurate
than information collected a year later. Most of the measures did not statistically differ from EHC or Q-List
format, but the differences which were observed all favored the EHC format.
2. Methods and Data
This chapter describes the analysis performed to evaluate the SIPP-EHC instrument. The analysis compares
key estimates from SIPP-EHC to corresponding estimates from SIPP for a variety of topics. Where administrative records for a topic are available, the analysis also assesses the accuracy of reporting in SIPP-EHC
relative to SIPP.
The following section describes the split-sample design that underlies this evaluation of SIPP-EHC. The
subsequent section explains how the administrative records are used as standard for the surveys. The final
two sections describe the specific statistics that we calculate, describe the statistical tests that we perform,
and provide a guide to interpreting the tables presented in the report.
2011 SIPP-EHC and 2008 SIPP Samples
The comparison of SIPP-EHC and SIPP presented in this report is essentially a split-sample experiment.
The samples for these two surveys were created at the same time using the same process. Effectively, units
(addresses) were randomly assigned to either the SIPP-EHC treatment or to the SIPP treatment.
Adjustments were made to both of these samples before interviewing began. These post-sampling adjustments are easily accounted for in our analysis so that the estimates from each survey are for the same
population. The present analysis re-weights the 2008 SIPP units to account for difference in state composition
between the two surveys. The 2008 SIPP sample had been adjusted to make the sample state representative
for the 20 largest states.
The post-sampling sample restrictions imposed for 2011 SIPP-EHC were designed to produce a low-cost field
test sample that still adequately sampled participants in federal and state programs to permit an informative
test of the SIPP-EHC instrument. The sample was restricted in four ways. Units were included only from
(1) the unit frame, (2) the self-representing primary sampling units (PSU), (3) the low-income within PSU
stratum, and (4) 20 particular states. For this evaluation, these same restrictions have been applied to the
SIPP data to create a suitable benchmark sample.
Table 2.1: Persons
Survey
AR
SIPP
12,855
10,429
SIPP-EHC
7,435
5,327
Total
20,290
15,756
Neither sample is nationally representative, but both samples permit estimation for the same population.
Calendar years 2010 and 2011 (CY2010 and CY2011) are the reference periods for waves 1 and 2 of the 2011
and 2012 SIPP-EHC instrument. There are two elements of uncertainty introduced by the fact that the SIPP
has been going for nearly 3 years longer than the SIPP-EHC. The first is a possible difference in sample
compositions owing to the time elapsed between the rostering of SIPP and SIPP-EHC. The second is the
possibility of a selected sampling effect produced through attrition. The many more waves and longer
10
Chapter 2
11
time period provided more opportunities for attrition; if respondents do not attrit randomly, then this may
introduce a difference in the composition of the responding sample.
Table 2.2: Person-months in sample
CY 2010
CY 2011
Survey
AR
SIPP
117,543
97,198
SIPP-EHC
83,044
63,882
SIPP
109,055
90,199
SIPP-EHC
59,560
43,389
369,202
294,668
Total
The total number of sample persons included in the analysis is 20,290 (see Table 2.1), with 7,435 from SIPPEHC and 12,885 from SIPP. The total number of sample person-months is 369,202. In CY2010, SIPP-EHC had
83,044 while SIPP had 117,543 person-months. In CY2011, SIPP-EHC had 59,560 and SIPP had 109,055 person
months (see Table 2.2). The total number of sample person-years is 35,084. 26,200 (74.6 percent) of those
person-years are complete. In CY2010, SIPP-EHC had 6,969 person-years (with 98.7 percent complete years)
and SIPP had 12,014 (65.2 percent complete) person-years. In CY2011, SIPP-EHC had 4,996 person-years
(98.6 percent complete)and SIPP had 11,105 person-years (63.1 percent complete) (see Table 2.3).
In CY2010, SIPP-EHC had 2,596 household-years, and SIPP had 2,596 household-years. In CY2011, SIPP-EHC
had 3,452 household-years, and SIPP had 1,942 household-years (see Table 2.4). The total number of sample
household-years is 11,769.
Table 2.3: Person-years in sample
Ever in year
CY 2010
CY 2011
Total
All year
Survey
AR
Survey
AR
SIPP
12,014
9,839
7,385
6,180
SIPP-EHC
6,969
5,327
6,879
5,315
SIPP
11,105
9,106
7,010
5,834
SIPP-EHC
4,996
3,617
4,926
3,614
35,084
27,889
26,200
20,943
Methods and Data
12
Table 2.4: Household-years
Survey
CY 2010
CY 2011
SIPP
3,779
SIPP-EHC
2,596
SIPP
3,452
SIPP-EHC
1,942
Total
11,769
Administrative records as a standard for survey reports
The Census Bureau has access to person-month level administrative records for several SIPP-EHC topics
under consideration: Medicare, Medicaid, Supplemental Nutritional Assistance Program (SNAP), Temporary
Aid to Needy Families (TANF), Old-Age, Survivors, and Disability Insurance (OASDI), Supplemental
Security Income (SSI), and annual employment and earnings. These records provide a benchmark for survey
reports on these topics. The data available for each topic are also described in the relevant chapters.
Comparison between administrative records and the responses of both survey instruments provide a test of
the relative accuracy of SIPP-EHC to SIPP. Of the 12,885 sample-persons between SIPP-EHC and SIPP, 15,756
were successfully linked to administrative records, with 5,327 from SIPP-EHC and 10,429 from SIPP. Though
administrative records are not without error, this report treats the administrative records as true and refers
to observations where a survey response matches an administrative record as accurate. Sampling units were
randomly assigned to SIPP or SIPP-EHC, so any measurement error in the administrative records should be
uncorrelated with survey assignment and should not introduce bias in comparisons of misreporting between
surveys. The accuracy of survey reports is measured at the person, person-year, and person-month levels.
Each survey report is compared to administrative records, and then the rate of agreement between survey
response and administrative record is compared across SIPP and SIPP-EHC. This difference-in-difference
approach gauges the relative accuracy of SIPP-EHC and SIPP in a way that isolates differences not due to
sample composition. Difference in rates of item-nonresponse between the surveys is a very important caveat
for our comparisons of survey estimates, since item non-response is likely to be non-random.
Harmonization of SIPP, SIPP-EHC, and administrative data
The data from the SIPP-EHC field tests are unedited in the sense that only the raw question responses are
available. Consequently, the person month status indicators and other variables used in the analysis had to
be constructed from responses to multiple questions, sometimes across the interviews of multiple persons
within a household.
These field test data also have none of the missing data imputed. In order to make valid comparisons with
SIPP, imputed values in the SIPP data have been coded as missing for this analysis (except where otherwise
indicated). For some topics, analysis variables were constructed from unedited SIPP data. For other topics,
edited SIPP data were used, but imputed data were excluded. Sample person-months from either SIPP-EHC
or SIPP that correspond to a nonresponding (neither self nor proxy) adult (15+) in a responding household
are excluded from this analysis.
For the government program topics (SNAP, TANF, SSI, OASDI), the exclusion of missing SIPP data meant
excluding all but the first observed receipt amount for those who report participation. This is because pro-
Chapter 2
13
duction SIPP processing applied the first observed receipt amount to all subsequent months of participation.
These “imputations” have also been excluded from the present analysis.
Description of the analysis
The analysis presented in this report compares estimates of key population characteristics between SIPP-EHC
and SIPP and investigates the relative quality of microdata from the two surveys using administrative
records.
For all topics covered in the report, estimates of key population characteristics are presented for each survey
in each of CY2010 and CY2011. Virtually all of these estimates are means. Estimates are presented at the
person-month, person-year, and person levels as indicated. Standard errors of these means were clustered at
the person level.
Tests are presented of equality of means across surveys within the calendar year. In order to draw inferences
about improvements in data quality across waves of the field test, tests are also presented of the statistical
significance of apparent differences across years in the within-year differences across surveys. All of the
statistical tests that we perform on medians and means and the difference between medians and means are
5% two-tailed t-tests using the asymptotic distribution of the t-statistics. We reject null hypotheses based on
t-tests greater than 1.96. Standard errors for medians are calculated via the woodruff method using SAS proc
surveymeans.
For topics with available administrative records, estimates based on administrative records are presented
alongside the corresponding estimates from the survey data. The survey based measures are presented
for just the linked sample for added comparability. We measure agreement between the two sources of
information for a given topic by calculating the mean absolute deviation (MAD) between the reports and
records. The mean absolute deviation is calculated as
N
∑i=L1 |sit − ait |
NL
where si and ai are the survey report and administrative record, respectively, and NL is the number of
observations in the linked sample. This measure has the advantage (over simply differencing the mean
status in the two sources) of not masking offsetting disagreements across person-months.
For observations where reported status and recorded status agree, we calculate, where applicable, the
mean absolute deviation between the survey-reported and administratively recorded amount of benefit or
earnings.
For participation indicators, the analysis also decomposes these mean absolute deviations measures into the
two types of possible errors: false negatives and false positives. We classify each person-month report in the
data as false negative (FN), false positive (FP), true negative (TN), or true positive (TP) based on agreement
between the survey report and the corresponding measure in the administrative records. We classify a report
about participation in a particular program as false negative if the administrative records for the program
indicate that the sample person participated in the program in a particular month, but the corresponding
survey report disagrees. The rate of FN is the proportion of false negative reports among person-month
observations with participants recorded in administrative data. The rate of FP is the proportion of false
positive reports among the survey-reports of participants.
For topics with person-month indicators, the analysis investigates the possibility that reports about events
further in the past may be less precise in the SIPP-EHC with its one-year reference period. Two possible
issues are considered: straight-lining and reverse-telescoping. Straight-lining refers to the possibility that
14
Methods and Data
respondents simply project backwards through the reference period whatever status they report in a more
recent reference month. A possible reason for this is that respondents may consciously mitigate the interview
burden that they would otherwise incur by being more detailed in their reporting. Reverse telescoping refers
to the possibility that respondent memory of events further in the past may simply be less precise.
The possibility of straight-lining or reverse-telescoping in SIPP-EHC data is evaluated by investigating the
pattern of reporting errors across calendar months. The analysis tests for equality across surveys and within
calendar year of the differences in the FN (FP) rates between the last four months of each calendar year
and 1) the FN (FP) rate in January and 2) the FN rate for the first four months of the calendar year. The
last four months of the calendar year are used as the benchmark since the reference period in the previous
design was four months long. We also test for statistically significant changes across calendar years in the
difference across surveys in the difference across these two periods of the calendar year in the rate of FN (FP)
reporting.
When no administrative records are available for a topic, a similar analysis is performed on the monthly
status indicators in which we look for more similar estimates at the end of a year than at the beginning. We
test the equality across surveys and within calendar year of differences in the mean reported status between
the last four months of each calendar year and the mean reported status for 1) January of the calendar year
and 2) the first four months of the calendar year.
Description of table elements
This section describes tables presented in the rest of the report.
Results for SIPP are from a sub-sample from 2008 SIPP that is comparable to the SIPP-EHC sample which is
not representative of the U.S. population. Results are weighted only to adjust for over-sampling of states in
2008 SIPP relative to 2011 and 2012 SIPP-EHC. 1
Columns labeled “Survey” present statistics based on survey reports for all in-universe observations for
which the relevant measure is non-missing. Columns labeled “Linked” present statistics based on survey reports for the observations in the corresponding “Survey” sample that were successfully linked to
administrative records. Columns labeled “AR” present statistics based on administrative data for the observations in the linked sample. Columns labeled “MAD” present the mean absolute deviation between the
survey-reported and administratively recorded measures for the linked sample.
Rows labeled “t-stat” present t-statistics from tests of equality between the SIPP and the SIPP-EHC. Rows
labeled “Diff-in-diff t-statistic” present t-statistics from tests of whether the difference between SIPP and
SIPP-EHC in CY2010 is equal to the difference between SIPP and SIPP-EHC in CY2011. Rows labeled
“Person-month observations,” “Person-year observations,” or “Person observations” present counts of
observations with no missing data.
Columns labeled “NIU” present the proportion of the sample person-months or person-years that are
not-in-universe for the indicated variable.
Household level estimates are presented for households as constituted in December of the indicated calendar
year for SIPP and as of the interview date for SIPP-EHC.
Rows labeled Test 1 and Test 2 present t-statistics of tests of equality of the indicated rates between the last
four months of each calendar year and 1) the rate in January (Test 1) or the rate for the first four months of the
1 Source: U.S. Census Bureau, Survey of Income and Program Participation, 2008 Panel. Survey of Income and Program Participation
˘
˘ S EHC) Field Test internal files, 2011 Panel. For information on sampling and nonsampling error
âA¸S Event History Calendar (SIPP âA¸
see http://www.census.gov/sipp/source.html
Chapter 2
15
calendar year (Test 2). Rows labeled “Pooled” present the indicated rate for all calendar months combined.
Columns labeled “Diff” present t-statistics for tests of difference in difference in the indicated rate across the
surveys across the calendar years.
3. Demographics
This chapter presents comparisons of reporting about demographics in SIPP-EHC and SIPP. The following
section describes the demographic data that are employed in the analysis discussed in the subsequent
section.
Description of demographics data
The data sources for the tables in this chapter are 2008 SIPP, 2011 SIPP-EHC, and 2012 SIPP-EHC. See
Chapter 2 for discussion of the SIPP-EHC and SIPP samples and weighting.
Person-year analysis variables of demographic characteristics are created for this analysis, including age,
sex, marital status, race, and Hispanic origin. All sample-persons are in universe for these variable. Samplepersons age 15 years or older are in universe for the marital status variable. Questions about demographic
characteristics are asked using the same wording in both years of SIPP-EHC and in the 2008 SIPP.
The 2011 and 2012 SIPP-EHC variables are created as of the interview month. The 2008 SIPP variables are
created from the edited data, dropping all imputations. Because the reference year includes three interview
points, person-year level data are taken from the latest wave where a report for the item is not missing. This
differs from the SIPP-EHC for which data is only taken from the time of interview.
Age is broken into five age categories: under 15, 15 to 24, 25 to 44, 45 to 64, and 65 and above.
Race and origin are combined into five categories: White alone (non-Hispanic), Black alone (non-Hispanic),
Asian alone (non-Hispanic), other (which includes those that choose the some other single race category and
those that include more than one race but are not Hispanic), and Hispanic (of any race).
Discussion of demographics results
This section presents and discusses the analysis of the demographic variables in SIPP-EHC.
In SIPP-EHC compared to SIPP, the proportion of sample-persons reported to be under age 15 is significantly,
but not substantially, lower in CY2010 (see Table 3.1). In the same calendar year, the proportion of samplepersons reported to be ages 25 to 44 is significantly, but not substantially, higher in SIPP-EHC than in
SIPP.
In CY2011, the proportion of sample-persons reported to be male is significantly, but not substantially, higher
in SIPP-EHC than in SIPP (see Table 3.2).
In CY2010 and CY2011, SIPP-EHC compared to SIPP reports fewer non-Hispanic whites and more Hispanics
(of any race) (see Table 3.3). The proportion of sample-persons reported to be non-Hispanic black is slightly
lower in SIPP-EHC than in SIPP in CY2010.
In SIPP-EHC compared to SIPP, the proportion of sample-persons reported to have a spouse absent is
significantly, but not substantially, higher in both calendar years (see Table 3.4). In CY2011, the proportion of
sample-persons reported to be married is significantly, but not substantially, higher in SIPP-EHC than in SIPP.
SIPP-EHC compared to SIPP reports fewer sample-persons as separated or never married in CY2011.
For race, there is lower item non-response in SIPP-EHC than in SIPP for both CY2010 and CY2011 (see
Table 3.5). For age, there is a statistically significant, but not substantial, difference in item non-response
between the two surveys in both calendar years. For sex, there is a significant, but not substantial, difference
in item non-response in CY2011.
16
Chapter 3
17
Table 3.1: Age
Under 15
15–24
25–44
45–64
65 and
above
SIPP
0.25
0.16
0.26
0.22
0.11
SIPP-EHC
0.24
0.17
0.28
0.22
0.10
t-statistic
2.14
1.14
2.71
0.51
1.67
SIPP
0.25
0.16
0.26
0.22
0.11
SIPP-EHC
0.24
0.15
0.26
0.23
0.12
t-statistic
0.45
0.76
0.53
1.13
0.71
Diff-in-diff t-statistic
2.17
2.27
4.21
2.45
3.64
35,046
35,046
35,046
35,046
35,046
CY 2010
CY 2011
Person-years
Table 3.2: Sex
Male
SIPP
0.47
SIPP-EHC
0.49
t-statistic
3.08
SIPP
0.46
SIPP-EHC
0.48
t-statistic
1.83
Diff-in-diff t-statistic
1.47
CY 2010
CY 2011
Person-years
35,064
Table 3.3: Race
White
Black
Asian
Other
nonhispanic
Hispanic
SIPP
0.28
0.29
0.05
0.03
0.35
SIPP-EHC
0.22
0.26
0.05
0.03
0.44
t-statistic
9.54
4.42
0.78
1.81
11.20
SIPP
0.28
0.27
0.05
0.03
0.36
SIPP-EHC
0.22
0.26
0.06
0.03
0.43
t-statistic
8.47
1.56
1.70
0.19
7.92
Diff-in-diff t-statistic
0.01
3.80
1.81
2.05
3.26
34,086
34,086
34,086
34,086
34,086
CY 2010
CY 2011
Person-years
Demographics
18
Table 3.4: Marital status
Married
Spouse
absent
Widowed
Divorced
Separated
Never
married
SIPP
0.34
0.02
0.06
0.11
0.04
0.43
SIPP-EHC
0.34
0.03
0.06
0.11
0.04
0.43
t-statistic
0.25
5.91
1.73
0.16
0.54
0.72
SIPP
0.35
0.01
0.06
0.10
0.04
0.43
SIPP-EHC
0.37
0.03
0.06
0.11
0.03
0.39
t-statistic
2.08
6.15
0.67
1.66
2.42
3.89
Diff-in-diff t-statistic
3.41
0.99
1.22
2.03
2.00
4.41
26,230
26,230
26,230
26,230
26,230
26,230
CY 2010
CY 2011
Person-years
Table 3.5: Demographic item-nonresponse rates
NIU (race)
Race
Sex
Age
NIU
(marital
status)
Marital
status
SIPP
0.00
0.05
0.00
0.00
0.25
0.01
SIPP-EHC
0.00
0.00
0.00
0.00
0.24
0.01
t-statistic
0.00
21.86
1.73
3.55
2.20
0.14
SIPP
0.00
0.05
0.00
0.00
0.25
0.01
SIPP-EHC
0.00
0.00
0.00
0.00
0.24
0.01
t-statistic
0.00
18.75
4.13
3.95
0.53
0.97
Diff-in-diff t-statistic
0.00
1.88
3.45
1.13
2.12
0.87
35,084
35,084
35,084
35,084
35,084
26,507
CY 2010
CY 2011
Person-years
4. Asset Ownership
This chapter presents comparisons of asset ownership rates and asset amounts between SIPP-EHC and SIPP.
The following section describes the asset data that were employed and the subsequent section discusses the
results.
Description of assets data
The data sources for the tables in this section are 2008 SIPP, 2011 SIPP-EHC, and 2012 SIPP-EHC. See Chapter
2 for discussion of the SIPP-EHC and SIPP samples and weighting.
Variables specific to asset ownership are created from the survey sources. The variables include ownership
indicators for interest-bearing assets (interest-earning checking account, savings account, money market
account, and certificates of deposit), stocks and mutual funds, and retirement accounts (401(k), Thrift, Keogh,
and IRA accounts). They also include the total balance of retirement savings accounts, the value of home,
and the value of first mortgage.
Sample-persons 15 years of age or older are in-universe for asset ownership variables. Ownership indicator
variables indicate ownership of having at least one element of the corresponding asset group. For example, a
sample-person with a reported stock and a missing report for mutual fund ownership is coded as an asset
owner for the mutual funds asset ownership indicator. Respondents are classified as having ‘no ownership’
for a particular asset group if they do not own any of the assets within that group. Sample-persons with
any missing report and no reported ownership in a given asset group are coded as missing. For example, a
respondent who refuses to answer the question about ownership of stocks and reports having no mutual
funds is classified as ‘no response.’ Sample-persons with no missing reports and no reported ownership of
assets in an asset group are coded as non-owners for the given asset group.
Total balance of retirement savings accounts refers to the total amount in 401(k), Thrift, Keogh, and IRA
accounts. The universe for this variable is anyone who reports having one of these accounts during the
reference period. The value of home and the value of first mortgage are both household-level variables. The
universe for the value of home is households whose primary residence is owned or being bought and whose
primary residence is not a mobile home. The universe for the value of first mortgage is households who
have at least one mortgage on their primary residence that is owned or being bought and whose primary
residence is not a mobile home. All values are collected as of the last day of the reference period.
An amount response is considered valid if the sample-person has a report of all amounts for the items he
or she was in-universe. For example, a sample-person that has a reported amount for the value of a 401(k)
account but has a reported refusal to the question asking about the value of an IRA account is coded as
missing. Similarly, a sample-person who has a reported refusal to the question about whether there is any
debt against his or her home is coded as missing, even though the sample-person does not have a directly
reported refusal to the question about the value of that debt. In the case of retirement accounts, a response
of ‘don’t know’ or ‘refuse’ to a question that sets the universe for the amount question or to the amount
question itself results in the respondent being coded as missing. If a respondent has multiple retirement
accounts and provided amounts for each, the values are added to produce the total value of retirement
accounts.
For 2011 and 2012 SIPP-EHC, the reference period was 1 year. The last day of the reference period for both
was December 31 of the corresponding calendar year. For SIPP, the data came from the Core and the Wealth
Topical Module, administered in Waves 7 and 10 of 2008 SIPP. The reference period for SIPP was 4 months,
19
20
Asset Ownership
which means that, depending on a particular respondent’s rotation group, for each calendar year the end of
the reference period ranged from August to November. All imputations were dropped from SIPP data and
only unedited data were used.
All amounts are collected as of the last day of the reference period, which, in the case of SIPP-EHC, always
refers to December 31 of the corresponding reference year. However, in the case of 2008 SIPP, the end of the
reference period could range from August to November. This means that amounts might not be directly
comparable across the surveys if balances in retirement accounts, home values, and values of mortgages
changed between August and December.
Assets amount questions are also placed differently in the question order in the SIPP-EHC and SIPP surveys.
In SIPP, the value of retirement accounts, the value of home, and the value of first mortgage are collected in
the Wealth Topical Module, which is administered at the end of the interview. This means that a respondent
encounters asset content twice (once in the Core and once in the Topical Module) and gets to the Topical
Module only upon successful completion of the entire Core instrument. In SIPP-EHC, a respondent is asked
about the assets content only once. It is unclear what impact, if any, these differences might have on item
non-response rates for assets items or on respondents report the value of a particular asset.
The number of questions a respondent needs to answer in order to be in-universe for questions about the
value of an asset also differs between SIPP and SIPP-EHC. In SIPP-EHC, respondents are asked two questions
about ownership of retirement accounts. The first asks whether a respondent had an IRA / Keogh account at
any point during the reference period. The second asks whether a respondent had a 401(k) / Thrift account.
Those who report ownership are then asked about the value as of the last day of the reference period. If a
respondent no longer owned such an account, the FR is instructed to enter the value of 0.
In 2008 SIPP, global ownership questions are asked in Core SIPP. Respondents are asked three, rather than
two, questions, since information about IRA and Keogh accounts is collected separately. If respondents get
to the Topical Module content, they are asked whether they still had the account for which they reported
ownership as of the last day of the reference period. Only those who answer in the affirmative are asked
about amounts. This means that in addition to differences in the placement of the asset content in the survey,
in SIPP respondents answer more questions about these assets.
For SIPP-EHC, sample-persons are in-universe for questions in the assets section if they are 15 years or older
as of the time of the interview, which did not take place until the spring of the calendar year following the
reference year. For the 2008 SIPP, respondents are in-universe if they are 15 years or older as of the last
day of the reference period, which ranges from August to December depending on respondent’s rotation
group.
Discussion of assets results
This section presents and discusses the analysis of asset ownership rates and amounts in SIPP and SIPP-EHC.
The table elements are described in Chapter 2, and, where appropriate, in table notes.
Some ownership rates for interest-earning assets and retirement savings accounts are lower in SIPP-EHC
relative to SIPP (see Table 4.1). However, even though these differences are statistically significant, they are
also quantitatively small. For example, in 2010 the ownership rates for interest-earning assets were 0.36 for
SIPP and 0.34 for SIPP-EHC; for retirement savings accounts, the ownership rates were 0.17 for SIPP and
0.14 for SIPP-EHC. There is no statistically significant difference in ownership rates of stocks and mutual
funds (see Table 4.1). There is no statistically significant difference in how SIPP and SIPP-EHC compare over
the two calendar years.
Between SIPP-EHC and SIPP, the mean value of retirement savings does not differ statistically in either
Chapter 4
21
Table 4.1: Asset ownership rates
Interestbearing
assets
Stocks and
mutual funds
Retirement
savings
SIPP
0.36
0.05
0.17
SIPP-EHC
0.34
0.05
0.14
t-statistic
2.13
0.22
4.63
SIPP
0.39
0.04
0.17
SIPP-EHC
0.35
0.04
0.14
t-statistic
3.66
0.37
4.55
Diff-in-diff t-statistic
1.77
0.17
0.35
21,952
22,015
21,996
CY 2010
CY 2011
Person-years
CY2010 or CY2011 (see Table 4.2). The 25th percetnile value is lower for SIPP-EHC in both years. The
SIPP-EHC also has lower median and 75th percentiles values in CY2010.
Table 4.2: Value of retirement savings
Mean
25th
percentile
Median
75th
percentile
SIPP
$91,606
$21,381
$63,202
$115,032
SIPP-EHC
$65,603
$3,980
$15,667
$49,917
t-statistic
1.42
3.89
5.02
3.42
SIPP
$131,446
$18,812
$68,907
$149,599
SIPP-EHC
$53,902
$4,932
$18,000
$56,500
t-statistic
2.26
4.01
3.52
6.66
Diff-in-diff t-statistic
1.32
5.53
1.43
1.43
Person-years
775
775
775
775
CY 2010
CY 2011
The mean home values are lower in SIPP-EHC than in SIPP in CY2010 and CY2011 (see Tables 4.3). SIPP-EHC
has lower estimates for the median and 75th percentile home value in CY2011.
In SIPP-EHC compared to SIPP, the mean value of the first mortgage is lower in both CY2010 and CY2011
(see Table 4.4). The 25th and 75th percentile values of first mortgage are lower in SIPP-EHC than in SIPP in
both years. The median value is lower in SIPP-EHC in CY2011; the median values do not differ statistically
between the surveys in CY2010.
For global ownership questions, SIPP-EHC has a slightly higher nonresponse rate relative to SIPP for
interest-bearing assets and retirement accounts in 2010 (see Table 4.5). However, the differences are not
quantitatively significant. There is no statistically significant difference in nonresponse rates for stocks and
mutual stocks. In 2011, there are no statistically significant differences for any of the assets.
Asset Ownership
22
Table 4.3: Home value
Mean
25th
percentile
Median
75th
percentile
SIPP
$197,050
$74,654
$125,154
$248,710
SIPP-EHC
$163,561
$66,250
$114,038
$196,992
t-statistic
2.97
1.82
0.09
1.11
SIPP
$184,189
$69,981
$119,295
$223,898
SIPP-EHC
$147,297
$59,250
$99,464
$174,333
t-statistic
3.25
0.19
2.92
3.30
Diff-in-diff t-statistic
0.21
0.39
0.82
0.09
Person-years
2,843
2,843
2,843
2,843
CY 2010
CY 2011
Table 4.4: Value of first mortgage
Mean
25th
percentile
Median
75th
percentile
SIPP
$145,933
$59,996
$100,374
$198,870
SIPP-EHC
$118,699
$42,250
$89,500
$158,500
t-statistic
2.88
3.51
1.38
3.11
SIPP
$148,232
$59,094
$99,378
$190,061
SIPP-EHC
$111,465
$41,875
$79,500
$145,313
t-statistic
2.87
3.07
2.96
2.67
Diff-in-diff t-statistic
0.59
0.07
0.87
0.21
Person-years
1,510
1,510
1,510
1,510
CY 2010
CY 2011
For asset amounts, SIPP-EHC has lower non-response rates relative to SIPP in both 2010 and 2011.
Chapter 4
23
Table 4.5: Asset ownership and amounts item-nonresponse rates
NIU (ownership)
Interestbearing
assets
Stocks and
mutual
funds
Retirement
savings
Amount
of
retirement
savings
Value of
home
Value of
first
mortgage
SIPP
0.39
0.02
0.02
0.02
0.91
0.31
0.39
SIPP-EHC
0.32
0.03
0.02
0.02
0.53
0.20
0.30
t-statistic
10.71
2.10
0.07
1.98
17.57
5.75
3.06
SIPP
0.39
0.02
0.02
0.02
0.90
0.31
0.37
SIPP-EHC
0.25
0.02
0.01
0.02
0.49
0.19
0.26
t-statistic
18.19
2.50
0.58
1.41
17.08
5.81
3.89
Diff-in-diff t-statistic
9.34
0.40
0.38
0.35
1.21
0.36
0.80
35,084
22,444
22,444
22,446
2,856
3,887
2,317
CY 2010
CY 2011
Person-years
5. Child Support
This chapter presents comparisons of reporting about child support receipt between SIPP-EHC and SIPP.
The following section describes the child support data that are employed in the analysis discussed in the
subsequent section.
Description of child support data
The data sources for the tables in this chapter are 2008 SIPP, 2011 SIPP-EHC and 2012 SIPP-EHC. See
Chapter 2 for discussion of the SIPP-EHC and SIPP samples. This section discusses the creation of analysis
variables specific to child support. There are two child support-specific variables: an indicator of receipt of
child support and amount of child support for those who report receiving child support.
The universe for receipt of child support is household members who are at least 15 years old and who are
the parent or legal guardian of a child under the age of 21 and the child(ren)’s other biological parent is not
a household member. All household members under the age of 15 are not in universe. This definition is
consistent for all three survey sources.
In all three survey sources, respondents are asked if they received any kind of financial support payments
from their child’s other parent. In the 2008 SIPP, imputed responses to this question are recoded to missing.
After reporting receipt of child support, the respondent is then asked the amount received in child support.
Only sample persons for whom child support receipt was reported are in-universe for a child support
amount.
Discussion of child support results
This section presents and discusses the analysis of reporting in SIPP-EHC of child support. The table
elements are described in Chapter 2, and, where appropriate, in table notes.
Table 5.1: Child support receipt
Monthly
participation
Annual
participation
Months of
participation
SIPP
0.23
0.34
9.22
SIPP-EHC
0.14
0.16
10.59
t-statistic
4.50
6.63
3.15
SIPP
0.22
0.29
10.24
SIPP-EHC
0.15
0.17
10.68
t-statistic
3.58
4.61
0.96
Diff-in-diff t-statistic
0.53
1.78
1.54
31,420
2,053
492
CY 2010
CY 2011
Observations
Table 5.1 presents the rates of monthly and annual participation in receipt of child support, as well as the
mean number of months in which child support was received. In both CY2010 and CY2011, reported receipt
24
Chapter 5
25
Table 5.2: Monthly child support amounts
Benefits
CY 2010
SIPP
$406.55
SIPP-EHC
$375.86
t-statistic
0.57
SIPP
$373.78
SIPP-EHC
$396.44
t-statistic
0.40
Diff-in-diff t-statistic
1.09
Person-months
3,087
CY 2011
of child support is higher in SIPP than in SIPP-EHC. In CY2010, SIPP respondents report fewer months
of participation in child support receipt. The unit of observation for the monthly participation column is
person-months. The unit of observation for the annual participation column is person-years for persons with
a reported participation status for all months of the year. The unit of observation for the months participation
column is person-years for participants with a reported participation status for all months of the year.
The mean monthly amounts of child support received, among those who received child support, are not
significantly different between the SIPP and the SIPP-EHC (see Table 5.2).
The number of child support spells does not differ statistically between the SIPP and the SIPP-EHC (see
Table 5.3).
Table 5.3: Spells of child support receipt for the
years 2010–2011
Spells
SIPP
1.07
SIPP-EHC
1.07
t-statistic
0.01
Persons
492
There is no evidence of reverse telescoping (see Chapter 2) in the pattern of month-by-month child support
receipt rates in SIPP-EHC (see Table 5.4). The difference between rates of child support receipt between the
two surveys do not differ statistically for September through December than for January (Test 1) or January
through April (Test 2) (see the t-stat column values for Test 1 and Test 2 in Table 5.4).
Finally, item-nonresponse is considered in Table 5.5. Item-nonresponse of receipt and amount of receipt
among recipients is much higher in the SIPP than in the SIPP-EHC. Item-nonresponse of amount reported is
significantly, but not substantially, higher in the SIPP than in the SIPP-EHC.
Child Support
26
Table 5.4: Monthly child support receipt
CY 2010
CY 2011
Diff
SIPP
EHC
t-stat
SIPP
EHC
t-stat
Jan
0.22
0.14
4.06
0.22
0.15
3.32
0.44
Feb
0.23
0.14
4.16
0.22
0.15
3.11
0.73
Mar
0.24
0.14
4.56
0.21
0.15
2.93
1.28
Apr
0.24
0.14
4.60
0.22
0.15
3.17
1.10
May
0.24
0.14
4.64
0.22
0.15
3.10
1.20
Jun
0.24
0.14
4.43
0.22
0.15
3.24
0.88
Jul
0.24
0.14
4.51
0.23
0.15
3.43
0.75
Aug
0.23
0.14
4.27
0.23
0.15
3.70
0.24
Sep
0.23
0.14
4.13
0.23
0.15
3.65
0.17
Oct
0.23
0.14
4.01
0.23
0.14
3.74
0.01
Nov
0.21
0.15
3.28
0.23
0.15
3.53
0.55
Dec
0.21
0.15
3.14
0.23
0.15
3.74
0.90
Pooled
0.23
0.14
4.50
0.22
0.15
3.58
0.53
Test 1
0.23
0.64
0.47
0.79
0.28
0.84
0.84
Test 2
0.76
0.75
0.99
1.26
0.53
1.34
1.53
Table 5.5: Child support item-nonresponse rates
NIU
Receipt
Amount for
recipients
SIPP
0.88
0.32
0.88
SIPP-EHC
0.91
0.02
0.00
t-statistic
7.22
21.44
21.73
SIPP
0.88
0.31
0.86
SIPP-EHC
0.90
0.02
0.00
t-statistic
3.40
20.44
14.16
Diff-in-diff t-statistic
3.76
0.16
0.48
369,317
40,040
604
CY 2010
CY 2011
Person-months
6. Disability
This chapter presents comparisons of reporting of disability status in SIPP-EHC and SIPP. The following
section describes the disability status data that are employed in the analysis discussed in the subsequent
section.
Description of disability data
The data sources for the tables in this chapter are the 2008 SIPP, 2011 SIPP-EHC and 2012 SIPP-EHC. See
Chapter 2 for discussion of the SIPP-EHC and SIPP samples and weighting.
Disability status is defined as having any of six types of difficulties: vision, hearing, ambulatory, cognitive,
self-care, or independent-living. This definition was developed for the American Community Survey but
has since been used in a number of federal surveys. This core definition of disability has also been the chosen
standard for Department of Health and Human Services data collections as required by Section 4302 of the
Affordable Care Act.
In the 2008 SIPP, the six core disability questions were asked during the Medical Expenses and Utilization
of Health Care topical module, which was fielded during waves 4, 7, and 10. The wave 7 data is used to
generate an estimate for 2010, while the wave 10 data is used to generate an estimate for 2011. The six
disability questions in this module are asked only of people age 15 and older. No disability information was
collected for children under age 15.
In the 2011 and 2012 SIPP-EHC, data are collected on several types of disability, including the six core
questions. Disability status is collected as a current period measure and the survey does not attempt to
capture month-to-month dynamics in a respondent’s status. For children under age 5, only the vision and
hearing questions are asked; for children between 6 and 14, vision, hearing, ambulatory, cognitive, and
self-care questions are asked. All six questions are asked for people 15 years and older.
Table 6.1: Disability
Disabled
Hearing
Vision
Cognitive
SIPP
0.21
0.05
0.04
0.09
0.14
0.05
0.09
SIPP-EHC
0.21
0.05
0.05
0.09
0.13
0.04
0.08
t-statistic
0.12
0.83
3.19
0.27
2.25
2.67
3.63
SIPP
0.18
0.03
0.03
0.07
0.12
0.04
0.08
SIPP-EHC
0.22
0.05
0.05
0.09
0.14
0.05
0.09
t-statistic
5.31
3.50
5.56
4.58
2.59
2.55
1.74
Diff-in-diff t-statistic
5.74
2.75
2.35
4.93
5.25
5.16
5.44
22,243
22,263
22,263
22,243
22,247
22,247
22,244
CY 2010
CY 2011
Person-years
Ambulatory Self-care
Ind.
To make comparable estimates of disability status between the data sources for this analysis, only respondents
27
Disability
28
age 15 and older are considered in universe. In the SIPP, responses with imputed information are coded as
missing and excluded from the calculation of estimates.
Discussion of disability results
This section presents and discusses the analysis of disability reporting in SIPP-EHC. The table elements and
weighting are described in Chapter 2, and, where appropriate, in table notes.
For CY2010, there are no statistically significant differences between the SIPP and SIPP-EHC in the rates of
overall disability, hearing difficulty, or cognitive difficulty (see Table 6.1). Between the two surveys, there
are small differences in the estimates for vision difficulty, ambulatory difficulty, self-care difficulty, and
independent-living difficulty. For CY2011, except for the independent-living category for which reported
difficulty does not statistically differ between the two surveys, reported difficulty in each specific category is
one or two percentage points higher in SIPP-EHC than in SIPP. The overall rate of disability in CY2011 is
four percentage points higher in the SIPP-EHC (see Table 6.1).
Table 6.2: Disability item-nonresponse rates
NIU
Disabled
SIPP
0.25
0.15
SIPP-EHC
0.24
0.11
t-statistic
2.49
6.73
SIPP
0.25
0.24
SIPP-EHC
0.25
0.02
t-statistic
0.01
43.11
Diff-in-diff t-statistic
3.33
26.19
35,084
26,486
CY 2010
CY 2011
Person-years
The item-nonresponse rates for SIPP-EHC are consistently lower than rates for similar items in the SIPP
in both CY2010 and CY2011 (see Table 6.2). From CY2010 to CY2011, item-nonresponse for disability in
SIPP-EHC decreased relative to SIPP.
7. Education
This chapter compares reported educational attainment and school enrollment in SIPP-EHC and SIPP data.
The following section describes the data on educational attainment and school enrollment that are employed
in the analysis discussed in the subsequent section.
The data sources for the tables in this chapter are the 2008 SIPP, 2011 SIPP-EHC, and 2012 SIPP-EHC. See
Chapter 2 for discussion of the SIPP-EHC and SIPP samples and weighting.
Description of data on educational attainment
The universe for reporting about educational attainment in both SIPP and SIPP-EHC is adults age 15 and
over. Both SIPP and SIPP-EHC asked the same question on educational attainment: "What is the highest
level of school [sample person] has completed or the highest degree received?"
In SIPP, respondents were also asked whether sample persons attended a vocational, technical, trade, or
business school, and, if so, whether they received a diploma from the school. The edited educational
attainment measure was a recoded variable combining information on educational attainment and receipt of
vocational diplomas. Reports of having less than an associate’s degree but also a vocational certificate were
recoded to a vocational certificate category on the final educational attainment measure.
The response categories for educational attainment in SIPP-EHC differed from SIPP in several ways. First,
the SIPP-EHC did not include a category for vocational certificates. SIPP-EHC sample persons with some
college but no degree were divided into two groups: some college credit, but less than one year; and one or
more years of college, but no degree. Both SIPP and SIPP-EHC captured educational attainment during the
interview month only.
For this analysis, a recoding of educational attainment in SIPP and SIPP-EHC is performed to make the data
comparable across the two surveys. SIPP sample persons whose highest level of educational attainment was
a vocational degree are recoded into the following categories: less than high school, high school completion,
or some college but no degree. SIPP sample persons’ educational attainment for December is used because
attainment can change within the reference period. In SIPP-EHC, a) some college credit, but less than one
year, and b) one or more years of college, but no degree are collapsed into a single response category for some
college but no degree. In both SIPP and SIPP-EHC, response categories for specific grades of elementary,
middle, and high school are collapsed into a single less than high school category. Any imputed data about
educational attainment is recoded as missing for this analysis.
Description of data on school enrollment
SIPP measured school enrollment by asking if respondents were enrolled in each of the four reference
months. Therefore, the edited data already include an indicator for school enrollment for each month. In
SIPP-EHC, an indicator for monthly enrollment was created. If the value for the start and end month of an
enrollment spell had a value greater than or equal to one, then a respondent was coded as enrolled for that
month.
In SIPP-EHC, school enrollment information was collected for all sample persons aged 3 and over. In
SIPP, only adults aged 15 or over were in-universe for school enrollment questions. Only sample-persons
aged 15 or older are included in the analysis. Sample-persons with imputed values are excluded from this
analysis.
29
Education
30
Discussion of educational attainment and school enrollment results
In both CY2010 and CY2011, SIPP-EHC compared to SIPP shows higher rates of less than high school
completion and lower rates of high school completion (see Table 7.1). The differences between SIPP and
SIPP-EHC estimates of educational attainment are significant only for the less than high school and high
school completion levels.
Table 7.1: Educational attainment
| File Type | application/pdf |
| File Modified | 2013-12-04 |
| File Created | 2013-09-26 |