Response to OMB Questions

"Harris-Kojetin, Brian A." <[email protected]>

12/07/2009 07:13 PM

To	<[email protected]>
cc	<[email protected]>, <[email protected]>
Subject	call on ACS Methods Panel

Hi Susan,

Could we talk later this week about the ACS Methods Panel submission? I have a variety of questions about how the clearance will work as well as some specifics about individual projects. Friday is pretty open for me if that will work for you and your folks. The rest of this week is awfully packed, but Thursday between 10 and 11 may also work.

I’m attaching a public comment that I received from Brookings and would appreciate a response to. Would you please share this with Tori and anyone else you think appropriate as well. The public comments you received should have been loaded into ROCIS, but weren’t. Would you please have Tom upload those when he returns.

Thank you!

Brian

Brian A. Harris-Kojetin, Ph.D.

Statistical and Science Policy

Office of Management and Budget

725 17^th Street NW, Room 10201

Washington, DC 20503

Phone: 202-395-7314

FAX: 202-395-7245

December 15, 2009: Census Bureau response to OMB questions and comments provided by the Brookings Institution.

The 2010 ACS Content Test will consist of a national sample of 70,000 residential addresses in the contiguous United States. For the Content Test Follow-up reinterview, we expect to have 40,000 reinterviews across the test and control panels. Our main evaluation measure for analyzing data quality differences between question versions is gross difference rates (or net difference rates for some items) that provide a measure of inconsistency in reporting between responses from the original interview and responses in the reinterview. According to our minimal detectable difference calculations, our sample size provides us with enough statistical power (80%) to detect a difference in the net/gross difference rates of at least 2 percentage points between the control and test groups. The 2 percentage point detectable difference was designed around the veterans population (since some of our test items apply only to that small population) and the assumption of a 9% gross difference rate; thus, we expect to detect a smaller difference for items where we do not have to restrict the sample to veterans and where we expect a smaller net or gross difference rate (a net or gross difference rate of 50% is the local maximum for the variance).

As Brookings notes, some of the questions being tested are low prevalence items (i.e., Food Stamp recipiency, cash public assistance recipiency) and thus the sampling error will be quite large for these items. Because this test includes other items in addition to these low prevalence items, combined in one test panel, we are unable to oversample target populations for specific items using our test design. However, as mentioned above, we still anticipate being able to detect at least a 2 percentage point difference (if not less) in net or gross difference rates between control and test panels for public assistance and food stamps.

Lastly, the test was not designed to study differences in net or gross difference rates across panels by race/ethnicity breakdowns with statistical precision, as this was not a stated goal of the test. A substantial increase in the sample size would be required to produce results for all race/ethnicity breakdowns. However, the Census Bureau will be able to perform some analysis for population subclasses such as Hispanics and Blacks for the public assistance and food stamps question items. While comparisons of data quality for these subpopulations are not part of our selection criteria, we can provide these results for informational purposes. The main evaluation measure for analyzing data quality differences between the public assistance question versions will be net difference rates (NDR) to measure systematic response error and the main evaluation measure for the food stamps question will be gross difference rates (GDR) to measure simple response variance. To calculate the minimum detectable differences (MDD) between the GDRs and the NDRs for the control and test panels for these two items, restricted to the Hispanic and Black populations separately, we assumed a range of expected GDRs and NDRs (2% - 10%). We assumed a range of values since we do not have historical data quality measures for these two topics for these subpopulations.

Public Assistance

Based on the expected range of NDRs, our sample size provides us with enough statistical power (80%) to detect at a minimum a difference in the net difference rates between the control and test groups ranging from 1.3% to 2.7% for the black population and 1.2% to 2.6% for the Hispanic population. The MDD ranges calculated were designed around the non-imputed person level universe for the public assistance item (age 15 and older) and the restriction to the Hispanic and Black populations.

Food Stamps

Based on the expected range of GDRs, our sample size provides us with enough statistical power (80%) to detect at a minimum a difference in the gross difference rates between the control and test groups ranging from 1.3% to 2.7% for the black population and 1.3% to 2.8% for the Hispanic population. The MDD ranges calculated were designed around the non-imputed household level universe for food stamps and the restriction to households with Hispanic and Black householders.

Regarding the Census Bureau’s proposed question on parental place of birth to be included in the American Community Survey (ACS) Methods Panel Tests: We greatly appreciate your continued support for our efforts to develop additional data on the second generation.

The parental place of birth (PPOB) questions that will be included on the 2010 American Community Survey (ACS) Content Test Questionnaire are the result of several months of conception, design, testing, analysis, and review. Subject-matter experts and questionnaire design specialists from the Census Bureau as well as outside agencies have contributed their expertise during every step of this process as members of the PPOB Subcommittee. The PPOB questions – as with all new questions on the Content Test – have undergone thorough review and have been revised to address potential problems. The issues and concerns mentioned in the letter were discussed and addressed by the PPOB Subcommittee. The PPOB questions to be included on the questionnaire – including “In what country was your/this person’s father born?” and “In what country was your/this person’s mother born?” – were chosen for the following reasons:

 The results of cognitive testing showed no conceptual differences between the shorter version of the questions and a longer version similar in format to the one proposed by Brookings. Also, Spanish speakers preferred the simpler question format.

 Respondents from Island Areas who participated in the cognitive testing demonstrated little difficulty in understanding the shorter question format.

 The shorter question can be administered in all modes of the ACS (mail, CATI, and CAPI) without changing core wording in any one mode, which could affect the results.

 To increase the likelihood that respondents with parents born in Island Areas do not respond “United States” as their parents’ place of birth, directions will be included on the questionnaire that specifically give the Island Areas of Puerto Rico and Guam as examples (i.e., “Print name of country, or Puerto Rico, Guam, etc.”). The ACS Users Guide will also include directions for those answering by mail to include a specific Island Area and provides examples. Guides for telephone interviewers and field representatives will also include clarification wording to encourage respondents to report a specific Island Area.

 The shorter question uses less space on a long survey.

 The PPOB questions, with the same wording and response format, have been part of the monthly Current Population Survey questionnaire every year since 1994.