Questions about the 2013 SIPP EHC Field Test
Please provide the results from the recontact experiment in the 2012 SIPP-EHC. How are these results informing 2013 or 2014 implementation of the SIPP-EHC?
The locating experiment data returns from the address update form (AUF) and from the four runs through the National Change of Address (NCOA) database. In the experiment, the sample of interviewed Wave 1 (2011 SIPP-EHC) households was divided into 3 groups, with two-thirds receiving the AUF and half of those receiving a twenty dollar incentive, leaving one third with no AUF, and no incentive. All eligible (Wave 1 interviewed) households were sent for NCOA runs. Work is in progress to match the AUF and NCOA return data to the 2012 actual interview locations. This work is expected to be completed in early January. The results from these analyses will inform future field procedures, most likely suggesting that we can include "possible addresses" into the Case Management system for access during the Wave 2+ fieldwork. The tables below show the return results from both the AUF and NCOA compared for the presence of move information and the incentive effect in the AUF responses. The Wave 2 non-interview rate for interviewed households in Wave 1 is 35.6% and certainly suffered due to the ‘this is a test’ message, and the high poverty and highly mobile sample.
NOTE: THESE RESULTS ARE PRELIMINARY AND MAY CHANGE.
Experiment treatment groups:
Cumulative Cumulative
treatment Frequency Percent Frequency Percent
------------------------------------------------------------------------------------------
No AUF, No Incentive 1 862 33.20 862 33.20
AUF, No Incentive 2 870 33.51 1732 66.72
AUF, Incentive 3 864 33.28 2596 100.00
Responses to AUF by Incentive group: Significant differences in response likelihood for incentive group versus non-incentive. 49.5% vs. 40.3%
Incentive response (returned an AUF)
Frequency|
Percent |
Row Pct |
Col Pct |no |yes | Total
---------+--------+--------+
no | 519 | 351 | 870
| 29.93 | 20.24 | 50.17
| 59.66 | 40.34 |
| 54.35 | 45.06 |
---------+--------+--------+
yes | 436 | 428 | 864
| 25.14 | 24.68 | 49.83
| 50.46 | 49.54 |
| 45.65 | 54.94 |
---------+--------+--------+
Total 955 779 1734
55.07 44.93 100.00
Wave 2 households where the full household moved, regardless of Wave 2 interview status by any move data present in the NCOA for that control number (Individual movers will be evaluated in future runs). About 39.5% of whole household movers, had information available from the NCOA database. Approximately 19% of our Wave 1 households were identified as movers, regardless of whether a wave 2 interview was conducted. This is higher than the national annual mover rate because of our high-poverty sample, and somewhat lower than the 25% poverty mover rate we would expect due to some loss-to-follow-up that may not have been coded as move related and the absence of partial household moves from this total.
Table of fullhhmv by anyncoa
fullhhmv anyncoa
Frequency|
Percent |
Row Pct |
Col Pct | 0| 1| Total
---------+--------+--------+
0 | 2028 | 69 | 2097
| 78.12 | 2.66 | 80.78
| 96.71 | 3.29 |
| 87.04 | 25.94 |
---------+--------+--------+
1 | 302 | 197 | 499
| 11.63 | 7.59 | 19.22
| 60.52 | 39.48 |
| 12.96 | 74.06 |
---------+--------+--------+
Total 2330 266 2596
89.75 10.25 100.00
The information rate from the NCOA was approximately twice that from the AUF. However, not all AUF returns indicated a mover; some were just ‘nothing changed’ responses.
anyauf anyncoa
Frequency|
Percent |
Row Pct |
Col Pct | 0| 1| Total
---------+--------+--------+
0 | 2230 | 218 | 2448
| 85.90 | 8.40 | 94.30
| 91.09 | 8.91 |
| 95.71 | 81.95 |
---------+--------+--------+
1 | 100 | 48 | 148
| 3.85 | 1.85 | 5.70
| 67.57 | 32.43 |
| 4.29 | 18.05 |
---------+--------+--------+
Total 2330 266 2596
89.75 10.25 100.00
Whole household movers were less likely to return the AUF form than non-whole household movers.
fullhhmv response
Frequency|
Percent |
Row Pct |
Col Pct |no |yes | Total
---------+--------+--------+
0 | 679 | 689 | 1368
| 39.16 | 39.73 | 78.89
| 49.63 | 50.37 |
| 71.10 | 88.45 |
---------+--------+--------+
1 | 276 | 90 | 366
| 15.92 | 5.19 | 21.11
| 75.41 | 24.59 |
| 28.90 | 11.55 |
---------+--------+--------+
Total 955 779 1734
55.07 44.93 100.00
Among the full household movers the NCOA and AUF response patterns were different, with a greater distribution of the movers identified in the NCOA data.
anyauf anyncoa
Frequency|
Percent |
Row Pct |
Col Pct | 0| 1| Total
---------+--------+--------+
0 | 197 | 119 | 316
| 53.83 | 32.51 | 86.34
| 62.34 | 37.66 |
| 92.92 | 77.27 |
---------+--------+--------+
1 | 15 | 35 | 50
| 4.10 | 9.56 | 13.66
| 30.00 | 70.00 |
| 7.08 | 22.73 |
---------+--------+--------+
Total 212 154 366
57.92 42.08 100.00
The Wave 2 non-interviews would appear to be related to the incentive receipt, but currently the analysis does not reconcile mover related non-response, which gets resolved as interviews in the subsequent mover case. This may significantly affect the relationships in the non-response evaluation. Those analyses are in progress and results will be available in the future.
What information is available on the results from the 2012 SIPP-EHC field test, and what are the plans for presenting/publishing these results?
We are currently resolving a problem with duplicate observations in the 2012 SIPP-EHC data created by a development issue with the new processing system. This issue is currently delaying substantive evaluations of linked person-month data. We are reviewing our key indicators and working to develop a summary evaluation package that we can make available to stakeholders in advance of releasing data from the new panel. Several separate components (i.e. earnings, medicare reports, poverty, educational enrollment, and others) have been presented in conferences, and subject matter areas continue to evaluate the re-engineering. There are no plans for releasing public use versions of the test data files. Should outside researchers need to access the test data, arrangements can be made on a case-by-case basis through special sworn status access at Census HQ or the Census RDC's.
In A.16, it was noted that Field Division will propose procedures for the 2013 SIPP-EHC field activities, and that these are “evolving.” Please provide more information on what regions are using which procedures and how these will be evaluated.
The Field Division (FLD) regional offices all utilize similar methods to locate movers, though often apply them in different mixes. During the follow-up discussions after 2012 SIPP-EHC interviewing, we discussed the regions use of FastData (a system for retrieving address information), the post office, and in-person discussions with knowledgeable neighbors. Some of the regions start active locating processes on every address in the sample during the lead-up to interviewing, while others allow interviewers to try to make first contact at the prior address before starting locating activities. During the development of the 2012 SIPP-EHC instrument, we were able to add a component to the Case Management System that will enable us to embed additional addresses as prospective and historical addresses. To this point we have been able to incorporate the past interview addresses into Case Management, and we are working to develop processes for adding additional addresses from sources like the locating activities NCOA databases. These new procedures are not ready to be incorporated in the 2013 SIPP-EHC due to resource constraints associated with getting the instrument ready for the field. These changes will be in place before we interview with Wave 2 of the production panel in 2015. We are continuing to work with FLD to implement the Unified Tracking System (UTS) and the Contact History Instrument (CHI) to track contact and locating success by region, these paradata tools will also allow the survey team to monitor the cost and success of different efforts, and possibly modify and tailor those procedures during the interview period.
Please provide a schedule for when information from this 2013 field test will be available and the schedule for the 2014 production EHC detailing what information from this test will be available when and how it will inform the production instrument and implementation.
We are developing detailed schedules as part of the transition to the new survey management structure, however the final versions of these are not yet available. Initial data from the 2013 SIPP-EHC field test will be available for analysis in May or June, although the edit system is still in development and not all items will be available as edited data or available except internally. The 2014 SIPP panel will begin field interviewing in February of 2014 and complete interviewing at the end of May 2014. The data processing and review will continue through the summer with the goal of an initial edited production data release before the end of interviewing for Wave 2 in 2015. The results from evaluations of the data from the 2011, 2012, and 2013 tests will continue and will be used to inform procedures and refine the Wave 2 plus instruments (2015 and beyond). Specifically the feedback data used in 2013 is refined from 2012, and the evaluation of the 2013 data will provide the information necessary to finalize dependent interviewing in 2015.
Also please provide your plans to inform and consult with stakeholders regarding the results of the EHC field tests and plans for the 2014 production EHC.
As part of the survey management realignment, we are establishing a communications staff to work with the survey team and with SEHSD. We are working on several different avenues for communication with our stakeholders. These include: developing a more dynamic and informative website, scheduling a pre-2014 stakeholder meeting (late spring/early summer 2013) to present and discuss results of evaluations and plans for facilitating evaluation of the 2014 panel data, scheduling another ASA/SRM meeting, hosting information sessions at conferences (possibly at this coming PAA), and looking at options to begin biennial SIPP workshops similar to those held by NSFG.
The SIPP-EHC field tests have not included any tests of incentives so far. We would expect that any plans to use incentives in the production EHC would be based on carefully conducted incentive experiments. If Census is considering incentives for the production SIPP-EHC, it is not clear why these experiments have not been included in the field tests. Please provide more information on your plans for incentives going forward.
The 2008 panel included a significant incentive test. The results of that experiment indicate that the historical practice of utilizing discretionary incentives fared less well than both no incentive and a $20 non-discretionary incentive with respect to panel attrition. The results indicated a positive effect for the non-discretionary incentive over no inventive for the 2008 panel. The SIPP-EHC is a longer interview than the current 2008 panel interview and would most likely benefit from a non-discretionary incentive. We are evaluating the budget requirements for incorporating incentives in the 2014 SIPP panel, whether they should be used at all, and whether they should be used in all waves, or only to retain sample in later waves. The sample for our series of field tests is based only on high poverty strata. This sample may not have produced results from an incentive test that would be generalizable to a general sample. Even given this consideration it probably would have been prudent to include incentives during Wave 2 and Wave 3 interviewing as additional evaluation components. While incentives during interviewing have not been included in our evaluations, as mentioned in the response to item 1, incentives were used to promote the response to the address update cards. This incentive yielded a 10 percentage point improvement in the return rates for the cards. We are still in the process of evaluating whether this incentive generated a higher response rate in 2012.
Please update Table 1 to show the current status of administrative records availability through CY 2012. For state-level files, please indicate which states are providing which files for which years.
We completed and have written up an evaluation of SSI reporting in 2012 SIPP-EHC using CY2011 SSR records. Our other SIPP-EHC evaluation work since the last ASA/SRM have been intended to evaluate reported earnings in 2011 SIPP-EHC using CY2010 W2 records. Both of these studies currently exist only as internal reports.
With respect to administrative records related to 2012 SIPP-EHC (CY2011), we currently have access to the following:
SNAP (NY state); - TANF (NY state);
Medicaid (through 2011q1, 27 states); - Medicare;
OASDI; - SSI;
PIC/TRACS (HUD); - W2s (initial delivery only)
Several more states are in various stages of negotiations to provide regular annual delivery of SNAP, TANF, and other state data through 2014.
A revised table 1:
CY2009 CY2010 CY2011
OASDI Yes Yes Yes
SSI Yes Yes Yes
Medicare Yes Yes Yes
SNAP NY and TX NY and TX NY and TX?
Annual Earnings Yes Yes Yes (initial delivery)
Job Counts Yes Yes Yes
Medicaid (MSIS*) Yes Yes (40 states) through Q1 (27 states)
TANF NY NY NY
Housing Subsidies Yes Yes Yes
*sample only for some states.
Please provide a current sample loss table for the 2008 SIPP panel to provide context for the comparisons between the EHC and the 2008 SIPP panel.
Household sample loss by the end of Wave 11 was 42.8 percent. The household sample loss rate varied between regional offices from a high of 50.8 percent to a low of 34.9 percent. It should be noted that the 2008 Wave 1 sample loss continues throughout the panel to account for the differential sample loss between 2008 and earlier panels.
Household Sample Loss Rates for the 2008 Panel
Wave |
Sample Loss Rate |
1 |
19.2% |
2 |
25.8% |
3 |
28.8% |
4 |
32.3% |
5 |
33.3% |
6 |
35.4% |
7 |
37.4% |
8 |
38.0% |
9 |
39.8% |
10 |
42.0% |
11 |
42.8% |
The household sample loss rate for the 2008 Panel is calculated as follows:
Household Sample Loss =
[(Wave 1 Type As * Growth Factor) + Type As and Ds (Current Wave)] /
[Interviews (Current) + (Wave 1 Type As * Growth Factor) + Type As and Ds (Current Wave)]
Note: The growth factor is a rate applied to Wave 1 Type As to estimate the unobserved growth of non-response from spawned households created by movers that can occur after Wave 1.
In previous SIPP panels the household sample loss rate calculation included permanent sample loss. This is defined as a Type A for two consecutive waves and a Type D for three consecutive waves. These Type As and Type Ds were dropped and regarded as a permanent loss to the universe of eligible households for interviewing. In the 2001 Panel, it was decided that, with the exception of the Wave 1 Type As, all Type As and Type Ds will be considered eligible for interviewing during each wave. This process will continue during the 2008 Panel.
Household Sample Loss Rates for the1992, 1993, 1996, 2001, 2004, and 2008 Panels
Wave |
1992 Panel |
1993 Panel |
1996 Panel |
2001 Panel |
2004 Panel |
2008 Panel |
1 |
9.3 % |
8.9 % |
8.4 % |
13.3 % |
14.9 % |
19.2% |
2 |
14.6 % |
14.2 % |
14.5 % |
21.9 % |
21.9 % |
25.8% |
3 |
16.4 % |
16.2 % |
17.8 % |
24.7 % |
25.6 % |
28.8% |
4 |
18.0 % |
18.2 % |
20.9 % |
25.9 % |
27.6 % |
32.3% |
5 |
20.3 % |
20.2 % |
24.6 % |
27.5 % |
29.8 % |
33.3% |
6 |
21.6 % |
22.2 % |
27.4 % |
28.2 % |
31.2 % |
35.4% |
7 |
23.0 % |
24.3 % |
29.9 % |
28.9 % |
32.5 % |
37.4% |
8 |
24.7 % |
25.5 % |
31.3 % |
30.3 % |
33.1 % |
38.0% |
9 |
26.2 % |
26.9 % |
32.8 % |
31.9 % |
34.0 % |
39.8% |
10 |
26.6 % |
--- |
34.0 % |
--- |
35.5 % |
42.0% |
11 |
--- |
--- |
35.1 % |
--- |
36.9 % |
42.8% |
12 |
--- |
--- |
35.5 % |
--- |
36.6 % |
|
13 |
--- |
--- |
--- |
--- |
--- |
|
The household sample loss rates for the 1992, 1993, and 1996 Panels were calculated as follows:
Household Sample Loss =
[((Type As and Type Ds (Permanent))* Growth Factor) + Type As and Ds (Current Wave)] /
[Interviews (Current) + ((Type As and Ds (Permanent])*Growth Factor) + Type As and Ds (Current Wave)]
The household sample loss rates for the 2001, 2004, and 2008 Panels were calculated as follows:
Household Sample Loss =
[(Wave 1 Type As * Growth Factor) + Type As and Ds (Current Wave)] /
[Interviews (Current) + (Wave 1 Type As * Growth Factor) + Type As and Ds (Current Wave)]
| File Type | application/msword |
| Author | field005 |
| Last Modified By | long0339 |
| File Modified | 2012-11-29 |
| File Created | 2012-11-26 |