Field Test Plan -- CPS ASEC Measurement Error

The purpose of this letter is to inform you of our plans to conduct additional research under the generic clearance for questionnaire research (OMB clearance number 0607-0725). We plan to conduct a field test with respondent debriefing in March 2009. This field test is one phase of a larger project to evaluate an experimental questionnaire designed to reduce measurement error in the Current Population Survey (CPS) Annual Social and Economic Supplement (ASEC).

The CPS ASEC is administered in March and asks questions about the previous calendar year, and prior research suggests that the timing of the survey and the calendar year reference period are associated with measurement error, at least with regard to health insurance coverage and participation in government programs. Several studies were conducted over the past few years to develop and test out new ways of asking retrospective questions, focusing on the 15-month time frame between the beginning of the calendar year and March of the following year. The resulting redesigned instrument recently underwent cognitive testing (submitted to OMB in a letter dated May 8, 2008). No fatal flaws were detected and minor modifications were made to address issues identified in testing; a summary of results is attached.

The next stage in this line of research is a field test in March 2009, in preparation for a larger scale field test in March 2010 (for which a separate request will be submitted). The purpose of the 2009 field test is to further refine the instrument focusing on issues identified in cognitive testing), to assess the overall timing of questionnaire administration, and to assess the capacity of the instrument to capture coverage in less-than-straightforward situations that were not necessarily represented in large numbers in the cognitive test sample. These situations include:

1. households where members’ coverage has gone through some turnover in the last 15 months;

2. large households (four or more people);

3. complex households (e.g.: non-relatives, distant relatives, unmarried partners, blended families);

4. households where members have a mix of different coverage types (e.g.: husband has direct-purchase; wife and children have job-based; mother has Medicare); and

5. households where some members have multiple types of coverage.

A draft copy of the questionnaire is attached.

The 2009 field test interviews will be conducted by telephone by Census Bureau headquarters staff and interviewers from the Hagerstown Telephone Center from March 23-27, using a hard copy questionnaire. Respondent debriefing questions will be administered at the end of the survey questionnaire. A copy of the debriefing questions is attached. The sample will be a combination of random digit dial (RDD) sample generated by Census Bureau staff “seeded” with telephone numbers of individuals whose health insurance coverage status is known through administrative records, provided by the Centers for Medicare and Medicaid Services.

The “seeded” cases will provide an opportunity to explore the feasibility of using administrative records in the 2010 field test in a reverse record check study to assess reporting accuracy. In the current field test, we will document the steps in the acquisition and protection of the records with an eye toward larger scale implementation in 2010, and assess various aspects of the quality and usefulness of the records. This project will be submitted for formal review and approval by means of the Administrative Records Tracking System (ARTS). This review ensures that projects provide ample confidentiality and privacy controls, and are in compliance with Title 13 of the U.S. Code and existing data agreements.

During the data collection, daily meetings will be held with interviewers and any flaws in the instrument will be discussed. Repairs will be made where possible, and the revised version will then be retested. This process will continue iteratively until no more flaws are detected. Interviews will also be timed to assess average length of administration. We hope to complete 50-100 interviews, depending on the degree of changes to the questionnaire and the diversity of respondent characteristics.

Respondents will be informed that their involvement is voluntary and that the information they provide is confidential and will be seen only by employees involved in the research project. Because the pretest sample is not a probability nationally representative sample, little effort will be made to call back respondents; however, messages will be left on answering machines informing respondents of the purpose of the call, and offering them a toll-free number to call and complete the survey if they wish. Persons 15 years of age or older will be eligible to serve as the household respondent, answering questions for all household members.

We expect the interview and debriefing combined to last an average of 15 minutes per household. Thus the total estimated burden for this research is 25 hours.

The contact person for questions regarding data collection and study design is listed below:

Joanne Pascale

Center for Survey Methods Research

Statistical Research Division

U.S. Census Bureau

4600 Silver Hill Road

Washington, D.C. 20233

(301) 763-4920

[email protected]

Summary of Cognitive Testing Results

Cognitive Testing of Experimental Health Insurance Questions: In March through May of 2008, five Census Bureau staff members conducted 36 cognitive interviews examining a set of questions on health insurance designed to capture both current coverage status and coverage during the previous calendar year, as well as periods of being uninsured. Testing did not turn up any fatal flaws. However, there were three main issues that arose which could benefit from further testing. The first had to do with dependents. If a respondent reports they have health insurance, they are asked the general source of the coverage – employer, government or other. Some respondents who were dependents on someone else’s employer-based plan were a bit reluctant to choose "employer" as a source of coverage since the source was not actually their employer – but a parent’s or spouse’s – and so they chose “other.” They were then routed to a more detailed question to determine the source, which included the response option “parent or relative.” However, this response option did not include followup questions on which parent or relative was the policyholder, and whether the coverage was job-related or directly purchased. The questionnaire was modified to include three new followup questions to capture who their policyholder was, and whether the plan was job-related or directly purchased, but this path was developed too late in testing to evaluate. We also developed two alternatives for the initial question on source to encourage dependents to report job-based coverage as “employer” plans, but we were unable to test these alternatives on any respondents.

A second issue had to do with months of coverage. When a respondent reported coverage, the routine was to first determine plan type, then months of coverage (between January 1, 2007 and the date of interview), then whether other household members were also covered on the plan and, if so, whether they were covered in the same months as the initial enrollee. In the vast majority of cases, when multiple household members are covered by the same plan, they are covered during the same months. However, we did find that the questionnaire was problematic when the person for whom the coverage was initially reported was covered for a shorter time period than other household members also on the plan. This was particularly evident for Medicare, when the younger of an elderly couple was interviewed and reported starting coverage at some point during 2007. Then they reported their spouse as also being covered, but the spouse’s coverage began years prior. This meant the older spouse was, technically, covered in the “same” months as the initial, younger respondent, but also had additional months of coverage earlier in the year which could have been missed. Thus skips were modified and a new question was developed. In the new routine, if the initial enrollee was covered all 12 months the question for subsequent enrollees was changed to: “And was [NAME] also covered all 12 months of 2008?” And if the initial enrollee was covered less than 12 months the wording was: “What months of 2007 was [NAME] covered?” (with a response category for “same months as [initial enrollee]”). Again, this modification could not be evaluated. In this case, however, the question wording is almost identical to another item in the questionnaire which was tested and shown not to be problematic. Thus the only concern with the new item could be that it appears in a different context.

Lastly, one of the main goals of the test was to evaluate whether the revised questionnaire functions as intended for people who had transitions in coverage over the 15-months – either from one plan to another or on and off coverage. While we did have some respondents who transitioned, and the questionnaire worked smoothly, these respondents may not have been in sufficient numbers, with diverse enough circumstances, to reveal any flaws in the questionnaire.

Thus, on the whole we feel the questionnaire functioned well in a number of different circumstances, but we did find a few flaws and made repairs and these new questions have not yet been tested. We also note that respondents with certain problematic circumstances may not have fallen into our sample.