2012 ReSIPP CARI Section A CHANGES ACCEPTED

SUPPORTING STATEMENT

U.S. Department of Commerce

U.S. Census Bureau

2012 Survey of Income and Program Participation

Computer Audio Recorded Interviewing Field Test

OMB NUMBER: 0607-XXXX

A. Justification

1. Necessity of Information Collection

The U.S. Census Bureau requests authorization from the Office of Management and Budget (OMB) to conduct a Computer Audio Recorded Interviewing (CARI) technology field test using the 2012 Survey of Income and Program Participation Event History Calendar (SIPP-EHC) Field Test questionnaire.

Computer Audio Recorded Interviewing is a data collection method that captures audio along with response data during computer-assisted personal and telephone interviews (CAPI & CATI). A portion of each interview is recorded unobtrusively, with the respondent’s consent, and the sound file is returned with the response data to a central location. By reviewing the recorded portions of the interview, quality assurance (QA) analysts can evaluate the likelihood that the exchange between the field representative (FR) and respondent is authentic and follows critical survey protocol as defined by the sponsor and based on best practices.

The Census Bureau will conduct the SIPP-EHC CARI test using the 2012 SIPP-EHC automated instrument and computer-assisted personal interviewing (CAPI) in 6 of the 12 Regional Offices. The SIPP-EHC CARI test will be conducted simultaneously with the 2012 SIPP-EHC test. However, The the SIPP-EHC CARI questionnaire will have the recording capability in use during the interview. The only content change to the instrument is the addition of a consent question which will record the respondent’s permission to audio record responses. Additionally, approximately 25 specific questions are programmed for recording for each person’s interview. See Attachment A for the recorded interview questions. Attachment B contains the entire interview questions.

This is the second CARI field test conducted by the Census Bureau. The first CARI field test was used to conduct behavior coding for the 2010 American Community Survey Content Test in late 2010. The Census Bureau is conducting this test to determine if the deployment of CARI will have any significant impact on response rates and item level responses. Previous tests for CARI have proven the capability of the technology. Other tests have also been conducted on non-voluntary surveys and proved promising. However, it is important for the Census Bureau to obtain information on the impact of this technology on data quality indicators for voluntary demographic surveys. If the test proves successful, this technology would be a major asset for all programs using computer assisted personal and telephone modes of data collection to assist in meeting quality objectives.

The 2012 SIPP-EHC CARI test will be in the field at the same time as the 2012 SIPP-EHC test. Both tests will be conducted between May and June 2012. An independent sample was selected for the CARI test from the same frames and high-poverty stratum as the 2012 SIPP-EHC which will be followed for a wave 2 interview using the 2012 SIPP-EHC instrument. These two samples will be interviewed at the same time, by the same interviewers. The overlapping collection of the SIPP-EHC CARI sample will be available for methodological comparisons with the 2012 SIPP-EHC wave 2 sample. The separate samples were determined to be necessary to avoid potential contamination of the loss-to-follow-up evaluation for the wave 2 sample as a result of any respondent concerns over recording. Specific comparison of the wave 1 SIPP-EHC CARI sample and 2012 SIPP-EHC wave 2 sample will include substantive topic evaluations, examinations of the presence and nature of transitions reported in the data, and evaluation of patterns of non-response across surveys. The comparisons will also add to the information for evaluating the effect of feedback (dependent data) in the 2012 SIPP-EHC. The CARI technology will be implemented using the 2012 SIPP-EHC data collection instrument.

In addition to the actual recording capability, the CARI Interactive Data Access System has been developed as a monitoring system that allows for the analysis of audio and image files to be conducted immediately after completion and transmission of the interview. The system is an innovative, integrated, multifaceted monitoring system that features a configurable web-based interface for behavior-coding, quality assurance and coaching. The system assists in coding interviews for measuring question and interviewer performance and the interaction between interviewers and respondents.

The CARI test will be conducted in 6 Census Regional Offices at the same time as the 2012 SIPP-EHC test utilizing the same interviewers. Approximately 1,300 addresses (based on response and coverage estimates derived from the 2011 SIPP-EHC

field work) will be selected for the 2012 SIPP-EHC CARI test, which will yield approximately 900 interviewed households. We estimate that each household contains 2.1 people aged 15 and older, yielding approximately 1,890 person-level interviews in the field test. Interviews take one hour on average. The total annual burden for the 2012 SIPP-EHC CARI interviews will be 1,890¹ hours in FY 2012.

FY 2012.

The SIPP is authorized by Title 13, United States Code, Section 182.

2. Needs and Uses

The 2012 SIPP-EHC CARI field test instrument will be evaluated in several domains including field implementation issues and data quality vis-à-vis the SIPP 2011 and 2012 field test results. Household non-response rates and item non-response rates will be compared to the 2011 and 2012 SIPP tests. The primary focus will be to examine the impact recording has on the quality of data.

The SIPP-EHC CARI test will begin in May 2012 and cover events from the 2011 calendar year. The sample will be approximately 1,300 designated addresses in the high poverty sample stratum of the SIPP 2000 sample redesign sampling frame. The same PSUs (primary sampling units) will be used to draw the samples for both the 2012 CARI SIPP-EHC test and the 2012 SIPP-EHC test.

Interviewers conducting the 2012 SIPP-EHC CARI test will receive training through self-study documents. In addition, the same interviewers will be used in the 2012 SIPP-EHC CARI test and 2012 SIPP-EHC and the tests will be conducted at the same time in May and June 2012.

This evaluation focuses on the 2012 SIPP-EHC CARI test. The CARI system is a software application that seamlessly records the verbal exchange of pre-specified questions between the interviewer and the respondent without disrupting the normal interview process. After the interview, any recordings obtained are digitally stored and can be reviewed later for quality assurance (QA) purposes. The CARI system has many potential benefits including the detection of data fabrication, the ability to provide better feedback to interviewers on proper interviewing procedures, the identification of questionnaire problems, and the reduction of field reinterview costs.

During the 2012 SIPP-EHC CARI interview, twenty-five questions will be recorded throughout the interview. However, the EHC section of the questionnaire will not be recorded. The twenty-five questions were selected by subject matter analysts involved in the SIPP-EHC redesign project. The questions cover a wide range of topics and include questions related to:

Demographic Characteristics Annual Program Income Housing Conditions Asset Ownership Address Verification

Household Roster Fertility Health Care Utilization and Medical Expenses Health Status

Several analyses will be conducted to provide a better understanding of how the CARI system may affect the data gathered using the 2012 SIPP-EHC instrument. The estimates and item nonresponse rates will be calculated and compared within the CARI sample to see if there are significant differences. CARI cooperation rates will be calculated. Models will be constructed to determine if certain characteristics of the respondent, household, geographic location, or interviewer can be used to predict CARI consent propensity. Any changes in interview completion rates, refusal rates, and interview break-off rates will be monitored and analyzed. Interviewer effects on estimates will also be evaluated using measures such as CARI cooperation rates and interview length.

More specifically, this project will focus on the following questions:

• Does CARI have a significant impact on the quality of the 2012 SIPP-EHC estimates of responses to CARI-specific questions?

The responses being recorded using CARI include several types of variables which require different methods of being analyzed. Estimates will be calculated for binary, multiple category, and continuous responses. Item nonresponse estimates will be calculated (don’t knows and refusals) for all variable types.

In addition to point estimates of means and proportions, variances, confidence limits, and coefficients of variation (standard deviation divided by the mean) will be calculated to conduct statistical testing and to assess statistical reliability. Prior to releasing the final report, the validity and reliability of all estimates, especially those produced at the RO level, will be evaluated.

For both binary and multiple category variables, proportions for each response can be calculated and compared between those individuals who consent to CARI and those who did not consent by using Fisher’s Exact test and/or Pearson’s Chi-Square test. The null hypotheses of both Fisher’s Exact test and Pearson’s Chi-Square test are the same for this data: the relative proportions of one variable are independent of a second variable. In this case, the null hypothesis will be that CARI consent is independent of response

All estimates, variances, and confidence limits produced will be calculated at the following levels:

• National level

• RO level

• Interviewer Experience level

Estimates will also be calculated at the RO and interviewer experience level but these estimates may not be reliable due to small sample size.

• Are there certain characteristics of respondents, geographic regions, or interviewers that can predict CARI consent propensity?

Person-level logistic regression models of CARI consent propensity scores will be created to evaluate which characteristics of the respondents or housing units affect CARI consent. Models will be fit at the national level and will be tested at the RO level. Models created at the RO level use smaller sample sizes and thus may not be as reliable. Evaluations of reliability will be made once the data is obtained.

Should time allow, multilevel and generalized linear mixed models (GLMMs) with interviewer characteristics included as random effects may be investigated. Using this type of model will allow the distribution of individual interviewer effects on CARI consent to be estimated. The primary purpose would be to estimate the amount of variance in CARI consent propensity is attributable to interviewers.

• What impact does CARI have on response rates, refusal rates, partial interview rates, interview break-off rates, or interview proxy rates?

Completion rates, partial interview completion rates, Type A refusal rates, and interview break-off rates will be calculated for all interviews in the 2012 SIPP-EHC CARI test and compared within the test data and to the 2011 and 2012 SIPP-EHC test data. In particular, interview break-off rates will be examined very closely. An interview break-off occurs when a respondent wishes to discontinue the interview before the interviewer has gathered enough data on the household to classify it as a sufficient partial. If interview break-off rates are higher for CARI interviews than for non-CARI interviews, this may have an impact on the quality of estimates.

• How willing are respondents in giving CARI consent?

Cooperation rates for CARI consent can be divided into three categories:

• CARI full cooperation rate

• CARI partial cooperation rate

• CARI refusal rate

• What will be used as indicators of interviewer performance?

Estimates will be produced for:

• Percentages/counts of consent responses for each interviewer as well as counts of mixed consent responses (i.e. households in which some respondents switch or turn off CARI consent).

• Percentages/counts of consent responses the interviewer obtained within households (i.e. the percentage/count of respondents within the same household in which the interviewer obtained full or partial CARI cooperation)

• Percentages/counts consent responses obtained among households (i.e. the percentage/count of households in which the interviewer obtained full or partial CARI cooperation)

• The length of the interview

If time allows, a longitudinal analysis of CARI cooperation rates over the length of the interview period may be done to determine if interviewers improve in their ability to gain respondent consent.

• When a respondent requests to turn off CARI consent during the interview, at what point (section) are they deciding not to be recorded? Does this lead to an interview break-off?

It may be that there are certain questions or sections of the interview that cause the respondent to reconsider their decision to be recorded. When and where this occurs in the interview would be of great interest in studying the effects of CARI.

The percentage of cases in which a respondent turns off CARI consent and decides to break off the interview will be calculated. The percentage of interview break-offs for households that gave full CARI cooperation, partial CARI cooperation, and refused CARI cooperation will also be calculated. These percentages will help in determining if CARI is a causal factor in the interview break-off.

Results from the 2012 field test will be used to inform and make final decisions regarding the implementation of CARI as a part of the quality assurance strategy for the SIPP instrument for production beginning in 2014 as well as other reimbursable demographic surveys. This OMB clearance request is for the 2012 SIPP-EHC CARI Field Test only.

3. Use of Information Technology

The survey is administered using CAPI methodologies. The Census Bureau field representatives (FRs) collect the data from respondents using laptop computers and the data are transmitted to the Census Bureau Headquarters via high-speed modems. Automation significantly enhances our efforts to collect high quality data with skip instructions programmed into the instrument and with information obtained in earlier interview segments fed back to the respondent. Response burden can be minimized by incorporating design features that make it easier to collect and record respondent information. Screening questions and lead-in questions are built into the automated instrument to skip respondents out of sections of the questionnaire that are not relevant or applicable.

The deployment of the CARI technology would be a major tool for all programs using computer personal and telephone modes of data collection to use to meet their quality objectives.

4. Efforts to Identify Duplication

The demographic data collected in the SIPP must be collected in conjunction with the labor force and program participation data in order for the information to be most useful; therefore, although we collect demographic data in conjunction with almost all surveys, we need to continue its present collection in the SIPP. There is no other current data source available that provides as comprehensive a set of statistics for analysis as described in question 2 above.

5. Minimizing Burden

The Census Bureau uses appropriate technology to keep respondent burden to a minimum. Examples of technology used to minimize respondent burden include: use of appropriate screening and lead in questions that serve to skip respondents out of sections of the CAPI instrument that are not relevant or applicable to them; use of flash cards to aid respondents with multiple response categories; and the arrangement of questions and sections in the CAPI instrument that facilitate the flow of administration from one topic area to another. Based on sponsor requirements related to interviewer critical performance behaviors, the CARI technology would reduce (but not eliminate) the need for additional data collection initiatives for purposes of obtaining data related to quality based on interviewer performance.

6. Less Frequent Collection

The 2012 SIPP-EHC CARI field test will be a one-time occurrence to test the impact of the CARI technology and household and item non-response.

7. Special Circumstances

There are no special circumstances associated with this clearance request.

8. Consultations Outside the Agency

The Census Bureau published a notice in the Federal Register on July 6, 2011,

Vol. 76, No. 129, page 39378, inviting public comment on our plans to submit this request. We received one comment from a citizen generally opposing the collection.

9. Paying Respondents

The Census Bureau does not plan to pay respondents during the 2012 SIPP-EHC CARI Field Test.

10. Assurance of Confidentiality

We are conducting this survey under the authority of Title 13, United States Code, Section 182. Section 9 of this law requires us to keep all information strictly confidential. The respondents will be informed of the confidentiality of their responses and that this is a voluntary survey by a letter from the Director of the Census Bureau that will be sent to all participants in the survey (Attachments C and D).

11. Justification for Sensitive Questions

The sources of income and assets are among the kinds of data collected and may be considered to be of a sensitive nature. The Census Bureau takes the position that the collection of these types of data is necessary for the analysis of important policy and program issues and has structured the questions to lessen their sensitivity.

12. Estimate of Respondent Burden

Based on our experience with the 2010 and 2011 SIPP Field Tests and in-house testing, the burden estimates for the FY 2012 SIPP-EHC CARI test are as follows:

2012 SIPP-EHC CARI TEST

FY 2012 BURDEN HOUR SUMMARY

	Respondents	Waves	Responses	Hours Per Response	Total Hours
Interview	1,890	1	1,890	1.0	1,890
Totals	1,890	1	1,890	1.0	1,890

We will obtain interviews from approximately 900 households, yielding approximately 1,890 individual interviews (2.1 individuals 15 years old or over per household).

The total number of burden hours requested for 2012 SIPP-EHC CARI Field Test interviews is 1,890.

13. Estimate of Cost Burden

There are no direct costs to respondents participating in the survey other than the time involved in answering the survey questions.

14. Cost to Federal Government

The production costs of all parts of this field test are approximately $1,100,000 in

FY 2012. That amount is included in the estimate of total costs to the federal government of the Census Bureau's current programs supplied to the OMB.

15. Reason for Change in Burden

The 2012 SIPP-EHC CARI Field Test is submitted as a new collection.

16. Project Schedule

The 2012 SIPP-EHC CARI Field Test advance letters will be mailed to sample addresses prior to interviewing. The 2012 SIPP-EHC CARI field test interviews will be conducted from May 2012 to June 2012. No public use data product will be released, however, the research and evaluation of the data will occur from July 1, 2012 to June 1, 2014. A field activity status report will be available in August 2012.

17. Request Not to Display Expiration Date

The expiration date is displayed in the advance letter that is sent to eligible households before the interview.

18. Exceptions to the Certification

There are no exceptions to the certification.

1See page 5 for a table on burden hours.